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FOREWORD 


In  the  early  nineties,  the  proposition  that  Advanced  Distributed  Simulation  (ADS)  was  the  wave 
of  the  future  for  T&E  was  advanced.  Reaction  was  mixed.  At  one  end  of  the  spectrum  were 
people  who  believed  the  need  for  live  testing  would  fade  away,  and  at  the  other  end  were  people 
who  scoffed  at  the  notion  that  ADS  had  any  utility  at  all  for  testers.  At  the  policy  making  level, 
expectations  were  high  and  skepticism  was  subdued.  At  the  implementation  level,  expectations 
were  low  and  skepticism  was  high. 

t 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT< &E)  program 
was  chartered  in  October  1994  to  conduct  an  objective  assessment  of  the  worth  of  ADS  for 
support  of  T&E.  While  the  Joint  Test  Force  (JTF)  work  is  not  complete,  they  have  shed 
considerable  light  on  issues  surrounding  ADS  use  for  T&E. 

The  JADS  data  suggests  the  reality  about  ADS  lies  between  the  extremes.  Expectations  that  ADS 
would  eliminate  or  drastically  reduce  live  testing  requirements  were  overblown.  Similarly, 
expectations  that  ADS  would  eliminate  the  need  for  stand-alone  models  and  simulations  were 
unrealistic.  On  the  other  hand,  expectations  that  ADS  was  essentially  useless  for  T&E  were 
clearly  wrong.  While  the  benefits  and  costs  of  ADS  are  very  test  program  specific,  the  JADS 
data  strongly  suggests  that  ADS  can  be  a  cost  effective  test  tool  in  many  applications. 

The  emerging  conclusion  of  the  JADS  JT&E  is  that  we  in  the  test  community  should  seriously 
consider  the  use  of  ADS  in  the  test  concept  development  and  planning  processes.  The  costs  and 
benefits  of ADS  will  vary  widely  from  program  to  program.  There  will  be  cases  where  it  should 
be  used,  and  cases  where  it  should  not.  The  ultimate  decision  to  use,  or  not  us e,  the  technology 
will  have  to  rest  upon  the  program  and  test  managers. 

We  believe  that  while  ADS  is  relatively  new,  it  can  be  a  valuable  addition  to  the  tester’s  tool  box. 
Our  hope  is  that  program  and  test  managers  will  routinely  include  consideration  of  ADS  use  in 
their  deliberations  and  planning  activities.  We  challenge  you  to  spend  the  intellectual  effort  to 
see  where,  how,  and  if  ADS  fits  into  your  programs.  We  encourage  you  to  examine  the  existing 
JADS  data,  identify  missing  data  and  work  with  JADS  to  identify  cost  effective  ways  of  using 
ADS  to  support  your  test  programs. 


Patricia  Sanders 
Director,  Test,  Systems 
Engineering  and  Evaluation 
OUSD(A&T) 


Philip  E.  Coyle 
Director 

Operational  Test  and  Evaluation 


EXECUTIVE  SUMMARY 


1.0  Overview 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  program  was 
chartered  by  the  Office  of  the  Secretary  of  Defense  in  October  1994  to  investigate  the  utility  of 
advanced  distributed  simulation  (ADS)  technologies  for  support  of  test  and  evaluation  (T&E). 
The  JADS  Joint  Test  Force  (JTF)  is  Air  Force  led,  with  Army  and  Navy  participation,  and  is 
scheduled  for  completion  in  1999.  This  report  addresses  the  first  of  three  separate  JADS  tests, 
the  System  Integration  Test  (SIT),  which  was  completed  in  October  1997. 

The  SIT  investigated  the  ability  of  ADS  to  support  air-to-air  missile  testing.  The  test  included 
two  sequential  phases,  a  Linked  Simulators  Phase  (LSP)  and  a  Live  Fly  Phase  (LFP).  Both 
phases  incorporated  one-versus-one  scenarios  based  upon  profiles  flown  during  live  test  activities 
and  limited  target  countermeasure  capability. 

The  LSP  distributed  architecture  incorporated  four  nodes:  the  shooter,  an  F/A-18  manned 
avionics  laboratory  at  China  Lake,  California;  the  target,  an  F-14  manned  avionics  laboratory  at 
Point  Mugu,  California;  a  hardware-in-the-loop  (HWIL)  missile  laboratory  at  China  Lake  which 
hosted  an  AIM-9M  missile;  and  a  test  control  center  initially  located  at  Point  Mugu  and  later  re¬ 
located  in  the  JADS  facility  in  Albuquerque,  New  Mexico. 

The  LFP  distributed  architecture  linked  two  live  F-16  aircraft  (a  shooter  and  target)  on  the  Eglin 
Air  Force  Base,  Florida,  Gulf  Test  Range;  the  Eglin  Central  Control  Facility;  an  HWIL  missile 
laboratory  at  Eglin  which  hosted  an  AIM- 120  missile;  and  a  test  monitoring  center  at  the  JADS 
facility  in  New  Mexico. 

This  report  describes  the  outcome  of  the  SIT,  the  conclusions  and  lessons  learned,  and  offers 
observations  on  the  implications  of  SIT  for  the  general  class  of  precision  guided  munitions. 

2.0  System  Integration  Test  Results  and  Conclusions 

Within  the  narrow  confines  of  the  SIT  data,  our  assessment  is  that  the  two  architectures  we 
employed  have  utility  for  support  of  T&E.  The  JADS  data  indicate  that  activities  ranging  from 
parametric  analyses  to  integrated  weapons  system  testing  are  both  practical  and  cost  effective. 
Our  broad  conclusions  and  lessons  learned  can  be  summarized  as  follows: 

-  For  T&E  applications,  the  technology  is  not  at  the  “plug-and-play”  stage.  While  practical 
and  cost  effective  in  many  cases,  implementation  is  more  challenging  than  many  people 
think.  Plan  for  a  lot  of  rehearsals  and  “fix”  time. 

-  The  effects  of  latency  and  other  ADS-induced  errors  can  often  (not  always)  be  mitigated. 

-  Synchronization  is  as  much  a  challenge  as  latency. 

-  Instrumentation  and  data  management  are  a  challenge. 

-  ADS  has  great  potential  as  a  T&E  support  tool:--It  is  a  valuable  addition  to  the  tester’s  tool 
kit.  ADS  will  not  obviate,  but  in  some  cases  it  may  reduce,  the  need  for  live  testing. 

-  Our  data  suggest  test  savings  are  possible. 
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3.0  Observations  for  Precision  Guided  Muntions  T&E 

Through  a  process  of  inductive  reasoning  we  can  transfer  some  of  the  SIT-based  specifics  to  the 
genera]  class  of  precision  guided  munitions  (PGM).  In  the  general  case,  the  elements  of  the  SIT 
architectures  are  basic  to  all  PGM  cases.  There  is  (1)  a  launch  platform  or  shooter,  (2)  a  PGM, 
(3)  an  intended  target,  (4)  an  operating  environment  (to  include  countermeasures),  and  (5)  a  test 
control  center. 

The  shooter,  PGM,  and  target  can  be  represented  in  any  of  the  three  forms  associated  with 
distributed  simulation:  live,  virtual,  or  constructive.  SIT  looked  at  an  AIM-9  and  an  AIM- 120. 
The  physical  dynamics  of  the  problem  are  comparable  with  any  class  of  PGM.  The  physics 
associated  with  detection,  tracking,  and  guidance  may  differ  significantly  depending  upon  bands, 
techniques,  and  the  operational  medium  a  missile  operates  in.  We  do  not  see  a  one-for-one 
transfer  of  SIT  techniques  to  other  tests.  Each  test  has  specific  requirements,  often  peculiar  to 
the  particular  system  under  test.  We  do  see  a  transfer  of  the  principles,  design  processes,  and 
methodologies  used  in  SIT. 

Countermeasures  were  only  represented  in  rudimentary  form  in  the  SIT,  but  we  see  no  technical 
impediments,  at  the  conceptual  level,  to  implementing  high-fidelity  countermeasures  in  ADS. 
The  devil  will  be  in  the  details,  and  costs  and  technical  challenges  will  be  very  case  specific. 
Complex  environmental  details  associated  with  atmospherics,  space,  oceanography,  etc.,  are 
more  challenging.  In  the  SIT  case,  the  LFP,  since  it  involved  flying  open  air,  incorporated  real 
atmospheric  effects. 

A  test  control  center  is  a  requirement  for  all  testing,  distributed  or  not.  Fortunately,  the  SIT 
experience  suggests  that  the  control  center  can  function  from  almost  anywhere.  The  inference  is 
that  an  existing  control  center  somewhere  may  well  meet  a  specific  tester’s  needs. 

The  SIT  program  was  budget  and  schedule  constrained.  Consequently,  there  were  important 
aspects  of  PGM  testing  which  SIT  did  not  explore.  From  a  single  shooter  perspective,  some  of 
these  included  multiple  launches  against  a  single  target,  single  launches  against  clustered  targets, 
and  multiple  launches  against  multiple  targets.  SIT  did  not  examine  few-on-few  or  many-on- 
many  scenarios.  Our  expectation,  unsupported  by  hard  data,  is  that  few-on-few  implementations 
are  possible.  The  difficulties  and  costs  would  be  extremely  sensitive  to  the  fidelity  requirements 
and  the  availability  of  existing  facilities,  e.g.,  HWIL  facilities  or  installed  system  test  facilities. 


The  SIT  results  suggest  strongly  that  ADS  has  good  potential  for  improving  PGM  testing.  The 
implication  is  that  test  planners  should  consider  the  technology  as  a  relevant  tool  for  their 
program  until  an  objective  assessment  suggests  otherwise.  Bottom  line:  Know  ADS  is  there, 
and  assess  how,  or  if,  it  should  be  used  in  a  specific  program. 
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1.0  Purpose  and  Background 

1.1  Report  Purpose 

This  report  summarizes  the  assessment  of  the  utility  of  advanced  distributed  simulation1  (ADS) 
for  the  test  and  evaluation  (T&E)  of  precision  guided  munitions  (PGM).  This  assessment  was 
based  on  the  results  and  lessons  learned  from  the  Joint  Advanced  Distributed  Simulation  (JADS) 
System  Integration  Test  (SIT)  testing,  along  with  results  from  other  related  efforts. 

This  report  also  considers  the  benefits  of  ADS-based  T&E  of  PGM  systems  including  Follow-On 
To  TOW  (FOTT),  Evolved  Sea  Sparrow  Missile  (ESSM),  ALM-9X,  AIM-120C,  Joint  Direct 
Attack  Munition  (JDAM),  High-speed  Anti-Radiation  Missile  (HARM),  Joint  Stand-Off 
Weapon  (JSOW),  Joint  Air-to-Surface  Standoff  Missile  (JASSM),  and  Standoff  Land  Attack 
Missile  -  Expanded  Response  (SLAM-ER).  Also  addressed  is  the  use  of  ADS  for  the  integration 
of  PGM  systems  to  advanced  launch  platforms  such  as  F-22,  F/A-18E/F,  Joint  Strike  Fighter 
(JSF),  and  Commanche. 

The  assessment  presented  in  this  report  gives  general  guidelines  for  implementation  of  ADS- 
based  testing  of  various  classes  of  PGM  using  various  ADS  architectures.  Detailed  requirements 
for  linking  specific  PGM  systems  using  specific  architectures  (other  than  the  AIM-9M  and  AIM- 
120  missiles  addressed  by  the  SIT)  were  not  developed.  The  extrapolation  of  SIT  results  to  PGM 
classes  other  than  air-to-air  missiles  is  based  on  informed  conjecture  without  rigorous  analysis  or 
supporting  data.  Applications  are  assumed  to  be  feasible  unless  there  is  evidence  to  the  contrary. 

1.2  JADS  Overview 

The  JADS  Joint  Test  and  Evaluation  (JT&E)  was  chartered  by  the  Deputy  Director,  Test, 
Systems  Engineering  and  Evaluation  (Test  and  Evaluation),  Office  of  the  Under  Secretary  of 
Defense  (Acquisition  and  Technology)  in  October  1994  to  investigate  the  utility  of  ADS 
technologies  for  support  of  T&E.  The  program  is  Air  Force  led,  with  Army  and  Navy 
participation,  and  is  scheduled  for  five  years. 

The  JADS  JT&E  is  directly  investigating  ADS  applications  in  three  slices  of  the  T&E  spectrum: 
the  System  Integration  Test  (SIT)  which  explores  ADS  support  of  PGM  testing,  an  End-To-End 
Test  (ETE)  which  explores  ADS  support  for  command,  control,  communications,  computers,  and 
intelligence  (C4I)  testing,  and  an  Electronic  Warfare  (EW)  Test  which  explores  ADS  support  for 
EW  testing.  The  JADS  Joint  Test  Force  (JTF)  is  also  chartered  to  observe,  or  to  participate  at  a 
modest  level  in,  ADS  activities  sponsored  and  conducted  by  other  agencies  in  an  effort  to 
broaden  conclusions  developed  in  the  three  dedicated  test  areas. 


'  ADS  is  a  networking  method  which  permits  the  linking  of  constructive  simulations  (digital  computer  models), 
virtual  simulations  (man-in-the-loop  or  hardware-in-the-loop  simulators),  and  live  players  located  at  distributed 
locations  into  a  single  scenario.  Such  linking  can  result  in  a  more  realistic,  safer,  and/or  more  detailed  evaluation  of 
the  system  under  test. 
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2.0  Supporting  Activities  and  Results 

2.1  System  Integration  Test  Overview 

The  SIT  evaluated  the  utility  of  using  ADS  to  support  cost-effective  testing  of  an  integrated 
missile  weapon/launch  aircraft  system  in  an  operationally  realistic  scenario.  The  purpose  of  the 
SIT  also  included  the  evaluation  of  the  capability  of  the  JADS  Test  Control  and  Analysis  Center 
(TCAC)  to  control  a  distributed  test  of  this  type  and  to  remotely  monitor  and  analyze  test  results. 

The  SIT  consisted  of  two  phases,  each  of  which  culminated  in  fully  linked  missions  simulating  a 
single  shooter  aircraft  launching  an  air-to-air  missile  against  a  single  target  aircraft.  In  the 
Linked  Simulators  Phase  (LSP),  the  shooter,  target,  and  missile  were  all  represented  by 
hardware-in-the-loop  (HWIL)  laboratories.  LSP  testing  was  completed  in  November  1996,  and 
results  were  documented  in  the  final  report  for  that  phase  [Ref.  1]  and  other  technical  papers 
[Refs.  2-4].  In  the  Live  Fly  Phase  (LFP),  the  shooter  and  target  were  represented  by  live  aircraft 
and  the  missile  by  an  HWIL  laboratory.  LFP  testing  was  completed  in  October  1997,  and  results 
were  documented  in  the  final  report  for  that  phase  [Ref.  5]  and  other  technical  papers  [Refs.  6-9], 

Missile  systems  selected  for  the  SIT  were  the  AIM-9  Sidewinder  for  the  LSP  and  the  AIM- 120 
Advanced  Medium  Range  Air-to-Air  Missile  (AMRAAM)  for  the  LFP. 

2.1.1  SIT  Linked  Simulators  Phase  Summary 

2. 1.1.1  LSP  Approach 

The  LSP  test  concept  was  to  replicate  a  previous  AIM-9M-8/9  live  fire  profile  in  an  ADS 
configuration  and  compare  missile  results  for  the  LSP  trials  to  those  from  the  live  fire  test.  The 
LSP  test  configuration  is  shown  in  Figure  1 . 

The  F/A-18  Weapon  System  Support  Facility  (WSSF)  at  China  Lake,  California  and  the  F-14D 
Weapon  System  Integration  Center  (WSIC)  at  Point  Mugu,  California,  were  the  shooter  and 
target,  respectively.  The  shooter  “fired”  the  AIM-9  in  the  Simulation  Laboratory  (SIMLAB) 
HWIL  facility  at  the  target  which  could  respond  with  countermeasures.  Runs  were  controlled 
from  a  test  control  center  which  ensured  all  nodes  were  ready  for  each  run,  issued  start/stop 
directions,  and  processed  data  packets  for  real  time  analysis  of  system  performance.  Test  control 
was  exercised  from  the  Battle  Management  Interoperability  Center  (BMIC)  at  Point  Mugu  while 
the  JADS  Joint  Test  Force  was  physically  relocating.  Control  switched  to  the  JADS  TCAC  in 
Albuquerque,  New  Mexico,  after  the  move  was  complete. 

Information  was  exchanged  among  participants  in  the  form  of  distributed  interactive  simulation 
(DIS)  protocol  data  units  (PDUs).  Entity  state  data  (positions,  velocities,  accelerations,  attitudes, 
and  attitude  rates)  at  the  output  node  were  converted  from  simulator  format  to  PDUs  and 
reconverted  at  the  receiving  end  into  simulator  format.  An  exception  was  the  link  between  the 
stores  management  system  (SMS)  of  the  shooter  and  the  missile  in  the  SIMLAB  which  used 
1553  data  bus  format. 
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Figure  1.  Linked  Simulators  Phase  Test  Configuration 


2. 1.1.2  LSP  Test  Results 

The  key  results  from  LSP  testing  were  as  follows: 

-  The  simulation  facilities  were  properly  linked,  and  the  missile  flyouts  were  valid  for  the 
target  representation  in  the  SIMLAB.  However,  this  target  representation  differed 
somewhat  from  the  target  data  originating  from  the  WSIC. 

-  The  manual  method  for  replicating  a  given  profile  resulted  in  very  good  run-to-run 
reproducibility  of  the  engagements. 

-  The  average  latency  of  all  entity  state  data  during  the  final  mission  was  relatively  small 
(<100  milliseconds  from  simulation  to  simulation)  and  consistent  run-to-run.  However, 
relatively  large  random  latency  variations  were  often  observed  which  resulted  in  an 
uncertainty  in  the  target  location  as  perceived  in  the  SIMLAB. 

-  The  ADS  network  provided  ample  bandwidth  and  no  loss  of  connectivity  during  testing. 

-  There  were  no  significant  ADS-induced  errors. 

-  The  reliability  of  the  long-haul  network  was  very  good,  and  the  availability  of  the 
complete  LSP  ADS  configuration  was  on  the  order  of  85%. 

-  Test  control  procedures  were  refined  throughout  the  preparation  process  and  worked  well 
during  testing. 
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2. 1.1.3  LSP  Lessons  Learned 

The  key  technical  lessons  learned  were  as  follows: 

-  Synchronization  of  activity  in  a  network  is  difficult  and  is  best  provided  by  a  “master 
clock”  built  into  the  architecture. 

-  Commonality  of  ADS  hardware  and  software  is  desirable. 

-  Understand  what  the  latency  requirements  are  before  designing  a  distributed  architecture. 
Incrementally  build  to  satisfy  them. 

Special  test  instrumentation  and  tools  are  required  to  support  distributed  testing.  The  tool 
set  must  support  rapid  identification  and  characterization  of  network  problems. 

The  key  infrastructure  and  process  lessons  learned  were  as  follows: 

ADS  requirements  must  be  developed  early,  understood  by  all  parties,  and  thoroughly 
documented. 

-  The  communications  required  to  exercise  test  control  must  be  identified  early. 

-  System  under  test  (SUT)  experts  must  be  involved  from  the  outset. 

The  architecture  build-up  must  be  incremental,  beginning  with  check  out  of  the  ADS 
elements  in  a  standalone  mode,  and  evolving,  step  by  step,  to  the  fully  integrated 
configuration. 

-  Problem  solving/fixes  frequently  require  verification  in  a  full-up  network  environment. 
Detailed  planning  for  data  management  is  a  necessary  precursor  to  testing. 

-  Contracting  to  support  ADS  should  most  often  be  on  a  cost  plus  basis  in  the  near  term. 
There  are  currently  too  many  unknowns  to  make  fixed  price  contracting  a  viable  option. 

-  Centralized  test  control  processes  have  to  be  integrated  with  established  local  processes 
and  procedures. 

-  Configuration  control  in  a  distributed  architecture  is  difficult,  but  essential. 

2. 1.1. 4  LSP  Conclusions 

The  LSP  ADS  configuration  has  utility  for 

-  Missile  weapon/launch  aircraft  system  integration  T&E. 

Parametric  studies,  due  to  good  pilot  manual  reproducibility  of  the  profiles. 

Rehearsal  and  refinement  of  live  engagement  scenarios. 

T&E  with  closed-loop  shooter/target  interactions  (e.g.,  prelaunch  tactics  evaluation). 

T&E  with  open-loop  missile/target  interactions  (e.g.,  the  missile  reacts  to  the  target,  but 
the  target  does  not  react  to  the  missile). 

The  LSP  ADS  configuration  has  potential  utility  for  closed-loop  missile/target  T&E  if  the  latency 
and  latency  variations  can  be  reduced  sufficiently  such  that  the  uncertainty  in  the  perception  of 
one  player’s  position  by  the  other  does  not  cause  disagreement  on  whether  or  not  the  target  was 

killed.  The  amount  of  acceptable  positional  error  is  a  function  of  the  design  of  the  test  and  the 
systems  involved. 

The  development  of  a  distributed  T&E  architecture  is  not  a  “plug-and-play”  exercise.  In  the  near 
term,  the  elements  available  for  linking  into  a  given  architecture  are  almost  certainly  not 
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designed  to  be  linked.  That  means  that  the  burden  for  making  linked  architectures  work  falls 
upon  the  interfacing  and  integrating  activities.  The  network  interface  units,  the  translation 
software,  the  geographical  transforms,  etc.,  are  the  interface  components  which  allow  distributed 
systems  to  function  with  existing  players  today. 

2.1.2  SIT  Live  Fly  Phase  Summary 

2. 1.2.1  LFP  Approach 

The  LFP  test  concept  was  to  replicate  previous  AMRAAM  live  fire  profiles  in  an  ADS 
configuration  and  compare  missile  results  from  the  LFP  trials  to  those  from  the  live  fire  tests.  In 
the  LFP,  ADS  techniques  were  used  to  link  two  live  F-16  aircraft  (flying  on  the  Gulf  Test  Range 
at  Eglin  Air  Force  Base,  Florida)  representing  the  shooter  and  target  to  an  AMRAAM  HWIL 
laboratory  (also  at  Eglin)  representing  the  missile.  This  configuration  allowed  data  from  live 
sources  to  drive  the  HWIL  laboratory  for  more  realistic  missile  results  and  is  shown  in  Figure  2. 


LIVE  AIRCRAFT 


MISSILE 


AASI  -  Advanced  Aircraft  Simulation  Interface 


CONTROL/COORD 
,  UMB  -  umbilical  RDL  -  rear  data  link 


Figure  2.  Live  Fly  Phase  Test  Configuration 


Global  positioning  system  (GPS)  and  telemetry  (TM)  data  were  downlinked  from  the  aircraft  and 
passed  to  the  Central  Control  Facility  (CCF)  at  Eglin.  GPS,  inertial  navigation  system  (INS),  and 
tracking  radar  data  for  each  aircraft  were  combined  by  the  TSPI  (time-space-position 
information)  Data  Processor  (TDP)  in  the  CCF  to  produce  optimal  entity  state  solutions.  The 
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aircraft  entity  state  data  were  transformed  into  DIS  PDUs  and  transferred  to  the  AMRAAM 
HWIL  simulation  at  the  Missile  Simulation  Laboratory  (MISILAB)  over  a  T3  link. 

The  shooter  aircraft  “fired”  the  AMRAAM  in  the  MISILAB  at  the  target  and  provided  data  link 
updates  of  the  target  position  and  velocity  to  the  missile  during  its  flyout.  The  AMRAAM  seeker 
was  mounted  on  a  flight  table  and  responded  to  radio  frequency  (RF)  sources  in  the  MISILAB 
which  simulated  the  seeker  return  from  the  target,  the  relative  motions  of  the  target  and  the 
missile,  and  electronic  countermeasures  (ECM).  A  link  between  the  CCF  and  the  JADS  TCAC 
allowed  JADS  personnel  to  monitor  and  record  the  simulated  intercepts. 

2.1.2.2  LFP  Test  Results 

The  key  results  from  LFP  testing  were  as  follows: 

-  The  live  aircraft  were  properly  linked  to  the  missile  HWIL  laboratory,  and  the  MISBLAB 
generated  valid  AMRAAM  data  during  the  engagement. 

-  The  TDP  generated  accurate  TSPI  solutions,  1-3  meters  in  position  and  1  meter  per 
second  in  velocity.  This  met  the  MISILAB  accuracy  requirements. 

The  shooter  and  target  TSPI  data  were  properly  synchronized  to  each  other  and  to  the 
umbilical  and  data  link  messages  for  input  to  the  MISILAB  simulation. 

Latencies  during  testing  were  relatively  stable  and  consistent,  but  fairly  large.  The  total 
latency  of  the  MISILAB  simulation  was  about  3.1  seconds.  This  large  value  of  latency 
was  due  to  the  processing  and  buffering  of  the  TSPI  data  to  produce  accurate  and  smooth 
solutions  and  to  the  synchronization  technique  used. 

The  ADS  network  provided  ample  bandwidth  and  no  loss  of  connectivity  during  testing. 

-  There  were  no  significant  ADS-induced  errors.  b 

Test  control  procedures  worked  well  during  testing  with  centralized  test  control  exercised 
from  the  CCF. 

2. 1.2.3  LFP  Lessons  Learned 

The  key  technical  lessons  learned  were  as  follows: 

Standalone  simulation  facilities  can  require  significant  (and  time  consuming)  software 
changes  before  effective  linking  is  possible. 

-  Live  fully  linked  risk  reduction  missions  were  needed  for  effective  integration  testing. 

-  Linking  may  require  special  purpose  interfaces,  and  their  development  must  be  factored 
into  test  planning. 

A  full  understanding  of  telemetry  limitations  was  needed. 

The  key  infrastructure  and  process  lessons  learned  were  as  follows: 

-  Ground  replays  provided  effective  rehearsals  and  were  helpful  for  troubleshooting. 

-  Existing  range  procedures  had  to  be  modified  for  ADS. 

A  premission  briefing  is  needed  before  each  mission,  as  it  was  critical  for  coordinating 
the  many  network  and  flight  test  issues.  ' 

Live  aircraft  operations  with  linked  facilities  required  more  contingency  planning  to 
quickly  decide  on  alternatives. 
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Live  aircraft  must  be  locally  controlled,  as  the  test  range  safety  policy  required  that 
tactical  control  of  aircraft  over  their  airspace  be  performed  at  the  Eglin  CCF. 

2. 1.2. 4  LFP  Conclusions 

The  LFP  ADS  configuration  has  utility  for: 

-  Missile  weapon/launch  aircraft  system  integration  T&E,  especially  evaluation  of  the 
targeting  messages  supplied  to  the  missile  by  the  shooter. 

Rehearsal  and  refinement  of  live  engagement  scenarios. 

-  Tactics  development  involving  closed-loop  interactions  between  the  shooter  and  target. 

-  Efficient  testing  utilizing  an  analyst-in-the-loop  for  timely  feedback  during  the  mission. 

This  test  architecture  would  not  support  testing  of  closed-loop  interactions  between  the  missile 
and  target  due  to  the  large  latencies  experienced.  However,  this  shortcoming  does  not  limit 
current  AMRAAM  testing,  because  essentially  all  live  fire  testing  involves  open-loop  scenarios 
in  which  the  target  drone  executes  scripted  (i.e.,  nonreactive)  profiles. 

2.2  Supporting  Studies  Summary 

In  addition  to  the  SIT  results,  other  studies  were  performed  both  inside  and  outside  of  the  JADS 
JTF.  The  findings  of  those  studies  which  are  relevant  to  PGM  ADS  applications  are  summarized 
as  follows: 

-  The  transfer  of  weapon-related  data  between  the  shooter  aircraft  and  the  missile  required 
a  dedicated  T1  link  for  the  LSP  architecture  [Ref.  10]. 

-  The  use  of  a  dedicated  T1  link  is  preferred  over  the  use  of  the  Defense  Simulation 
Internet  (DSI)  for  transferring  PDUs  between  a  remote  domed  flight  simulator  and  the 
SIMLAB  AIM-9M  HWIL  laboratory.  This  conclusion  was  based  on  data  latency 
requirements  [Ref.  10].  Also,  the  DSI  system  can  be  extremely  unreliable  [Ref.  11]. 

-  Linking  between  manned  flight  simulators  and  missile  HWIL  laboratories  requires 
latencies  of  less  than  100  milliseconds,  synchronization  of  the  simulators  using  absolute 
GPS  time,  and  the  use  of  dead  reckoning  algorithms  [Refs.  12-18].  Latency  requirements 
for  some  discrete  events  (e.g.,  missile  launch  and  detonation)  can  be  as  small  as  50 
milliseconds  [Ref.  18]. 

-  Latencies  between  linked  domed  manned  flight  simulators  have  been  measured  in  the 
range  of  200  to  700  milliseconds  [Refs.  19-21].  Such  latencies  limit  somewhat  the  use  of 
domed  simulators  for  evaluation  of  the  closed-loop  interaction  between  air-to-air  missiles 
and  target  aircraft  (see  Appendix  A). 

-  Departures  from  DIS  standards  may  be  necessary  in  live  applications  [Ref.  22].  Such 
departures  are  readily  accommodated  by  implementing  high  level  architecture  (HLA) 
concepts  [Ref.  23], 
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-  The  use  of  HLA2  overcomes  many  limitations  and  undesirable  features  of  DIS  standards 
by  allowing  data  protocol  flexibility,  eliminating  unnecessary  coordinate  transforms,  and 
reducing  the  amount  of  data  transmitted  between  nodes  by  only  transmitting  entity 
attributes  which  have  changed  [Ref.  23],  An  HLA-compliant  air  intercept  missile 
federation  (similar  in  architecture  to  the  SIT  LFP)  is  being  developed  by  the  Air  Force 
Developmental  Test  Center  (AFDTC)  under  Foundation  Initiative  2010  [Ref.  25]. 

ADS  has  been  used  to  support  virtual  prototyping  [Ref.  26]  and  surface-to-air  missile 
(SAM)  T&E  [Ref.  12].  Programs  are  underway  to  implement  ADS  for  virtual  PGM 
testing  [Ref.  27],  FOTT  T&E  [Ref.  28],  and  torpedo  testing  and  training  [Refs.  29,  30], 

3.0  Overall  Advanced  Distributed  Simulation  Utility  Assessment 

3.1  JADS  Issues 

The  JADS  JT&E  program  was  chartered  to  investigate  the  utility  of  ADS  for  both  developmental 
test  and  evaluation  (DT&E)  and  operational  test  and  evaluation  (OT&E).  The  charter  letter 
identifies  three  issues  to  be  addressed  [Ref.  31]: 

-  Investigate  the  present  utility  of  ADS,  including  DIS,  for  T&E.  The  utility  assessment 
includes  evaluating  the  validity  of  data  from  tests  using  ADS  and  the  benefits  of  using 
ADS  in  T&E. 

Identify  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E. 

-  Identify  the  requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to  support 
a  more  complete  T&E  capability  in  the  future. 

The  ability  of  ADS  to  support  PGM  T&E  will  be  assessed  in  terms  of  these  issues. 

3.2  General  Utility  of  ADS  for  Precision  Guided  Munitions  T&E 
3.2.1  General  Utility  Assessment 

ADS  has  utility  for  PGM  DT&E  and  OT&E  because  ADS-supported  tests  can  provide  valid 
PGM  performance  evaluations  in  a  number  of  areas  addressed  by  both  DT&E  and  OT&E  (see 
Section  3.2.3)  and  because  there  are  benefits  to  using  ADS  for  PGM  T&E  (see  Section  3  2  2) 
The  applications  for  a  specific  PGM  system  (see  Section  4.2)  depends  on  (1)  the  test  objectives 
(2)  the  characteristics  of  the  PGM,  (3)  the  availability  of  high-fidelity  PGM  digital  simulation 
models  (DSMs)  or  HWIL  laboratories,  and  (4)  the  details  of  the  test  scenario.  Guidelines  for 
implementing  ADS-supported  testing  for  a  specific  PGM  system  are  provided  at  Appendix  C. 


HLA  is  replacing  DIS  as  the  standard  protocol  for  linking  distributed  simulations  [Ref.  24],  HLA  uses  a  standard 
set  of  rules,  tools,  and  a  runtime  infrastructure  (RTI)  to  allow  simulations  to  send  and  receive  information.  HLA 
allows  the  simulations  to  treat  communications  in  a  more  abstract  manner  by  providing  a  common  interface 
specification  and  isolating  the  simulations  from  communications  protocol  implementations.  Because  simulations 
can  be  less  aware  of  each  other,  HLA  should  encourage  reuse  of  existing  models  for  different  applications.  There 
are  also  no  standard  data  formats  analogous  to  DIS  PDUs.  Users  can  form  their  own  messages  based  on  their  needs 
Likewise  users  can  use  PDUs  if  they  feel  that  the  PDUs  represent  the  data  necessary  for  their  simulation.  In  most 
cases,  the  conclusions  from  DIS-based  testing  should  directly  apply  to  HLA-based  distributed  testing. 
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3.2.2  General  ADS  Benefits 


The  benefits  of  ADS-supported  testing  are  best  realized  when  this  technique  is  added  to  a  total 
PGM  testing  program.  ADS-supported  tests  are  not  meant  to  replace  any  of  the  current  testing 
techniques,  including  live  fire  tests,  but  rather  to  supplement  current  techniques  and  provide  a 
more  comprehensive  evaluation  of  a  PGM  system.  When  the  appropriate  mix  of  testing 
techniques  is  used  (see  Section  3.2.3),  the  following  benefits  are  realized  from  the  addition  of 
ADS-supported  testing: 

-  Cost  savings  benefits. 

--  A  PGM  testing  program  which  uses  ADS-supported  tests  to  supplement  live  fire  tests 
can  be  more  cost  effective  than  live  fire  testing  alone.  In  a  limited  number  of  cases, 
relatively  inexpensive  ADS-supported  tests  can  replace  costly  live  tests.  Generally 
live  tests  are  not  replaced;  instead,  the  proper  use  of  ADS  can  result  in  a  higher 
success  rate  for  the  live  tests  by  identifying  failures  before  the  fact  (cost  avoidance) 
and  can  aid  in  the  optimal  selection  of  live  test  scenarios  and  associated  measures.  A 
methodology  for  determining  the  cost  benefit  of  mixing  ADS  missions  into  a  PGM 
testing  program  is  given  in  Appendix  B,  along  with  an  example  of  applying  the 
methodology  to  one  phase  of  AMRAAM  testing. 

-  Improved  testing  benefits. 

-  Testing  using  a  linked  laboratory  ADS  architecture  (similar  to  the  LSP  architecture)  is 
more  reproducible  than  live  fire  testing,  because  scenario  conditions  are  more  readily 
controlled  and  trials  can  be  replayed  for  additional  PGM  responses.  This  allows  more 
trials  to  be  combined  for  analysis,  giving  greater  confidence  in  evaluation  results. 

--  ADS-supported  testing  allows  the  evaluation  of  certain  classified  techniques  in  which 
the  ECM  device  cannot  be  permitted  to  radiate  its  RF  emission  on  an  open  range. 
Rather,  the  ECM  emissions  can  be  restricted  to  the  PGM  HWIL  laboratory  where  they 
are  screened  from  unauthorized  observation  and  where  the  effects  of  the  ECM  on 
PGM  performance  can  be  immediately  observed  by  analysts. 

--  ADS  allows  the  force  density  of  the  scenario  to  be  increased.  The  number  of  friendly 
and  threat  systems  can  be  increased  by  representing  them  with  either  manned 
laboratories  (if  realistic  man-in-the-loop  control  of  the  systems  is  needed)  or  DSMs  (if 
scripted  behavior  is  acceptable).  The  inability  to  evaluate  system  performance  in 
combat-representative  environments  is  a  common  limitation  in  OT&E  and  an  area  in 
which  ADS  can  improve  the  operational  test  (OT)  environment  [Ref.  32]. 

—  ADS-supported  tests  exhibit  more  realism  than  either  analytical  simulation  models 
(because  actual  hardware  is  used)  or  standalone  HWIL  laboratories  (because  realistic 
shooter  and  target  inputs  are  provided). 

More  efficient  testing  benefits. 

—  Testing  using  a  live  shooter-target  ADS  architecture  (similar  to  the  LFP  architecture) 
is  more  efficient  than  live  fire  testing  because  the  analysts  get  immediate  feedback  on 
each  pass  of  a  multiple  pass  mission.  This  allows  adjustments  to  be  made  to  the 
remaining  test  matrix,  if  necessary,  while  the  live  shooter  and  target  platforms  are  still 
on  range.  This  “analyst-in-the-loop”  feature  of  ADS  testing  would  be  especially 


9 


useful  in  efficiently  progressing  through  an  ECM  testing  matrix  which  involves 
varying  a  number  of  ECM-related  parameters. 

--  Live  fire  tests  can  be  realistically  rehearsed  using  ADS.  This  would  ensure  the  proper 
setup  of  the  scenario  and  reduce  wasted  live  fire  attempts  in  which  the  proper  scenario 
conditions  are  not  achieved.  This  use  of  ADS  would  also  reduce  the  risk  of  a  live  fire 
testing  program  by  identifying  scenarios  which  cannot  be  correctly  executed  or  which 
cannot  achieve  the  stated  objectives  [Ref.  32], 

3.2.3  General  Role  of  ADS  in  PGM  T&E 

As  stated  in  Reference  33,  The  goals  of  missile  system  T&E  are  achieved  through  the  proper  use 
of  all  the  means  of  test  and  analysis  at  hand.  The  most  desirable  T&E  approach  is  to  use  realistic 
flight  tests  (launches)  to  produce  the  required  technical  data.  However,  this  is  not  practical  for 
all  missile  systems.  The  unit  cost  of  the  missile,  the  cost  to  conduct  a  flight  test,  the  program 
schedule,  or  all  three  may  limit  the  number  of  flight  tests  that  can  be  performed.  In  some 
instances  the  desired  conditions  or  environment  can  only  be  represented  in  a  laboratory 
environment.  As  a  result,  other  test  techniques  must  be  developed  and  used  to  complement  the 
data  available  from  launches.”  The  techniques  used  for  PGM  T&E  are  described  in  Table  1 
(based  on  Table  2  from  Ref.  33). 


Table  1.  PGM  Performance  Evaluation  Techniques. 


Technique 

Description 

Analytical  Study 

Review  and  understanding  of  technical  documents,  drawings,  schematics,  and  other  materials 
to  the  extent  of  complete  understanding  of  the  system.  This  results  in  the  determination  of 
strong  and  weak  points,  assessment  of  potential  performance,  and  definition  of  the  specifics 
for  testing. 

Laboratory/Field 

Test 

Operation  of  the  system  or  subsystems  in  a  controlled  open-loop  laboratory/field  environment 
to  define  specification  data,  threshold  data,  and  basic  subsystem  performance 

1  Simulation  Techniques  f 

Analytical 

(DSM) 

Mathematical  representation  of  the  PGM  system/subsystems  in  closed-loop  operation.  The 
analytical  simulation  is  normally  programmed  on  a  digital  computer.  The  most  commonly 
used  analytical  simulations  are  (1)  the  trajectory  simulation,  and  (2)  the  lethality  simulation 

HWIL 

Laboratory 

Closed-loop  analytical/hardware  representation  of  the  PGM  system  used  to  evaluate  PGM 
performance  as  affected  by  guidance/seeker/signal  processing.  Sophistication  and  realism  are 
obtained  by  integrating  as  much  of  the  PGM  hardware  as  possible  in  place  of  analytical 
models.  Hardware  integration  provides  a  valid,  realistic,  and  complete  representation  of  the 
PGM  operation  that  cannot  be  achieved  in  analytical  simulations. 

ADS-Supported 

Test 

Uses  ADS  techniques  to  link  live  platforms,  HWIL  laboratories,  or  DSMs  representing  the 
shooter  and  target  to  an  HWIL  laboratory  or  DSM  representing  the  PGM.  Provides  more 
realistic  inputs  to  the  PGM  simulation,  resulting  in  a  more  valid,  realistic,  and  complete 
representation  of  the  PGM  operation  than  can  be  achieved  by  the  PGM  HWIL  laboratory  or 
DSM  in  standalone  operation. 

Flight  Test  Techniques 

Captive  Carry 
Tests 

Tests  where  the  PGM  is  carried  aloft  into  a  representative  environment  (i.e.,  live  tarsets, 
countermeasures,  chaff,  clutter).  The  tests  are  of  open-loop  confiauration 

Launches 

The  PGM  is  launched  in  a  representative  environment.  This  is  the  most  realistic  and  desirable 
of  the  data  sources. 
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Each  of  the  techniques  given  in  Table  1  has  a  role  in  supporting  PGM  T&E.  Table  2  (based  on 
Table  8  from  Ref.  33)  defines  which  testing  techniques  will  likely  be  used  to  determine  missile 
system  performance  as  a  function  of  missile  areas  of  performance.  As  Reference  33  states, 
“Taken  individually,  the  data  from  each  type  of  test  may  be  revealing.  However,  a  successful 
evaluation  is  dependent  on  the  appropriate  utilization  of  each  technique  as  defined  in  an 
integrated  missile-system  T&E  plan.” 

Note  that  the  unshaded  area  (i.e.,  traditional  unlinked  testing  techniques)  in  Table  2  shows  the 
following: 

-  No  single  testing  technique  is  the  primary  data  source  for  all  performance  areas. 

-  Traditionally,  analytical  (or  digital)  simulations  provide  the  primary  data  for  more 
performance  areas  than  any  other  technique. 

Although  live  fire  tests  provide  data  for  almost  all  performance  areas,  these  tests  are  the 
primary  data  sources  for  only  four  areas.  Note  that  the  primary  data  source  for  lethality 
assessments  is  analytical  simulations,  rather  than  live  fire  tests. 

In  Table  2,  ADS-supported  testing  is  divided  into  two  categories  depending  on  whether  the 
representation  of  the  PGM  system  is  by  a  DSM  or  by  an  HWIL.  For  either  of  these  two  PGM 
representations,  the  shooter  and  target  can  be  represented  by  either  live  platforms,  HWIL 
laboratories,  or  DSMs.  The  choice  of  shooter,  target,  and  PGM  representation  depends  on  the 
test  objectives,  the  details  of  the  scenario,  and  the  performance  area  and  type  of  PGM  being 
evaluated.  Guidelines  for  the  appropriate  ADS  architecture  will  be  discussed  for  specific  PGM 
types  in  Section  4. 

Table  2  was  constructed  from  Table  8  in  Reference  33  by  adding  the  shaded  ADS-supported 
testing  columns.  The  assumption  in  adding  these  columns  was  that  ADS  tests  using  a  DSM  to 
represent  the  PGM  would  be  able  to  evaluate  all  the  same  performance  areas  as  using  a  DSM 
alone  and  ADS  tests  using  an  HWIL  to  represent  the  PGM  would  be  able  to  evaluate  all  the  same 
performance  areas  as  using  an  HWIL  alone.  However,  testing  based  on  ADS  configurations  can 
allow  more  realistic  evaluations,  assuming  that  an  ADS  configuration  is  possible  and  can 
produce  valid  PGM  data,  and  ADS-supported  tests  would  be  the  preferred  source  of  primary 
evaluation  data  in  such  cases  (as  indicated  by  entries  with  “P*”  in  the  table).  Also,  there  are  some 
cases  in  which  the  ADS-supported  test  allows  performance  areas  to  be  evaluated  which  could  not 
be  evaluated  using  only  the  PGM  simulation  in  a  standalone  configuration.  An  example  is 
evaluation  of  fire  control  system  errors  using  an  LFP-type  architecture  (in  which  the  actual 
targeting  messages  from  the  live  shooter  tracking  the  live  target  are  input  into  the  PGM  HWIL 
laboratory).  This  linked  configuration  allows  the  missile  message  and  initial  conditions  accuracy 
to  be  evaluated,  as  shown  by  results  of  LFP  testing  [Ref.  5],  but  the  PGM  HWIL  laboratory  by 
itself  cannot  evaluate  this. 

Roles  for  ADS-supported  testing  can  be  identified  for  both  DT&E  (in  which  the  general 
objective  is  to  define  PGM  capabilities  using  preproduction  PGM  subsystems)  and  OT&E  (in 
which  the  general  objective  is  to  determine  the  PGM  system’s  operational  effectiveness  and 
suitability  using  production-level  PGM  systems  in  realistic  combat  scenarios). 


11 


Table  2.  Performance  Evaluation  Matrix. 


Performance  Area 


Launch  Acceptability  Redons 


Relative  Range  (F-Pole) 


Safe  Separation/Launch  Envelo 


No-Escapes  Zones 


Stability  and  Control 


Adaptive  Gain  Control 


Aerodynamic  Performance 


Body  Bending 


Propulsion  Management 


Warhead 


Formation  Target  Capability 


Multiple  Target  Capability 


ECCM  Capability 


Maneuvering  Targets 


Lookdown/Shootdown  Capability 
RF  Background 
IR/EO  Background 


Threat  Radar  Cross  Section 


Threat  Glint 


Threat  Signature  Polarization 


Threat  IR/EO  Emissions 


IR/EO  CCM  Capability 


Chaff 


Weather 


Inertial  Reference 


Midcourse  Guidance  Performance 


Acquisition  (Seeker  Range) 


Terminal  Guidance  Handover  and 
Search 


Lethality  (Probability  of  Kill) 


Weapon  System  Integration 
Safety  of  Operation 
Weapon  System  Operation 


Weapon  System  Assessment 
Single  Aircraft/Missile  System 
Tactical  Environment 


Fire  Control  System  Errors 
Missile  Message  Accuracy 
Initial  Conditions  Accuracy 


Software  Assessment 


Independent  Software  V&V 


=  primary  data  source  without  ADS  testing 
X  =  supporting  data  source 


P*  =  primary  data  source  when  ADS  testing  is  valid 


XX  X  "OX  XU  X  X  U  X  X  xxxxxx  xxxxxxxxxxxuxx 


DT&E  evaluation  areas  which  can  utilize  ADS  include  [Ref.  33] 

-  Evaluation  of  the  guidance  section  influence  on  PGM  system  performance.  Currently, 
this  evaluation  uses  the  PGM  HWIL  laboratory  in  a  standalone  configuration.  The  use  of 
ADS  will  result  in  more  realistic  evaluations. 

-  Definition  of  integrated  launcher/PGM  weapons  system  capabilities.  Currently,  this  is 
done  with  simulations  only.  The  use  of  ADS  can  result  in  much  better  system  integration 
evaluations.  High-fidelity  integration  can  begin  prior  to  having  flight/soldier-ready 
systems. 

-  Support  of  live  fire  tests. 

—  ADS  can  be  used  to  provide  realistic  test  rehearsals,  to  refine  test  scenarios,  and  to 
analyze  post-test  results. 

-  ADS  can  be  used  to  better  extrapolate  live  fire  results  to  different  scenarios. 

OT&E  evaluation  areas  which  can  utilize  ADS  include 

-  Evaluation  of  the  PGM  system  in  the  tactical  environment  with  representative  battlegroup 
tactics  and  realistic  threat  representations.  Currently,  cost  and  safety  constraints  limit  the 
density  of  forces  in  typical  OT&E  to  much  less  than  would  be  found  in  battle.  Also,  it  is 
usually  impossible  to  obtain  the  actual  threat  systems  that  the  PGM  would  operate  against 
and  have  them  operate  in  a  realistic  “unconstrained”  manner.  The  use  of  ADS  can  greatly 
augment  the  force  density  during  testing  with  either  manned  simulators  or  DSMs 
representing  the  additional  forces,  as  discussed  in  Section  3.2.2.  (DSMs  could  represent 
those  threat  systems  for  which  there  is  no  actual  hardware.) 

Support  of  live  fire  tests. 

--  ADS  can  be  used  to  provide  realistic  test  rehearsal,  including  refinement  of  live  fire 
scenarios  in  order  to  ensure  test  objectives  are  met. 

—  ADS, can  be  used  for  post-test  analysis.  In  particular,  ADS  can  aid  in  resolving 
anomalies  observed  in  live  fire  testing. 

--  ADS  can  be  used  to  supplement  live  fire  tests.  ADS  can  be  used  for  those  scenarios 
for  which  it  provides  valid  results,  along  with  live  fire  tests,  to  support  a  cost-effective 
total  testing  program  (see  Table  2  and  Appendix  B). 

3.3  Critical  Constraints,  Concerns,  and  Methodologies  for  PGM  T&E 

Certain  constraints  limit  the  applicability  of  ADS  to  PGM  T&E.  Some  of  these  constraints  are 
due  to  the  HWIL  facilities  or  DSMs  used  to  represent  the  PGM  and  are  common  constraints  for 
both  ADS-supported  testing  and  standalone  testing.  Other  constraints  result  from  linking 
requirements  and  are  related  to  ADS  implementation.  The  constraints  are  summarized  as 
follows: 

PGM  HWIL  limitations. 

--  Limited  fidelity  of  seeker  scenes  and  background. 

-  Restrictions  on  line-of-sight  rates  due  to  seeker  table  motion  limitations. 

-  Limited  seeker  field  of  view  due  to  seeker  scene  presentation. 

-  PGM  DSM  limitations. 

—  Limited  fidelity  of  PGM  representation  and  response  (since  hardware  is  modeled). 
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-  Only  DSMs  which  run  in  real  time  can  be  linked  to  live  platforms  or  manned 
simulators. 

ADS-related  constraints. 

--  Real-time  applications  using  the  linked  laboratory  ADS  architecture. 

—  Maximum  allowable  latency  restricted  to  100  milliseconds  for  highly  dynamic 
missile-fighter  aircraft  interactions  due  to  dead  reckoning  limitations.  The 
importance  of  certain  discrete  events  (e.g.,  missile  launch  and  detonation)  to  the 
engagement  fidelity  can  further  restrict  the  maximum  latency  to  50  milliseconds. 

—  Live  target-shooter  applications. 

—  Testing  closed-loop  interactions  between  a  simulated  missile  and  a  live  target 
aircraft  is  generally  not  possible  because  means  do  not  currently  exist  for  linking 
the  response  of  the  missile  to  the  live  target.  Techniques  are  needed  to  take  the 
missile  entity  state  and  seeker  emissions  data  output  from  the  HWIL  laboratory 
(or  DSM)  and  transmit  these  to  the  target.  Also,  the  target  needs  the  added 
capability  to  receive  the  transmissions  and  to  use  them  to  drive  detection/warning 
systems  and/or  to  drive  automatic  countermeasures  (CM)  responses. 

Current  range  safety  policy  requires  control  of  live  range  assets  to  be  exercised  at 
the  range.  While  this  may  be  acceptable  when  only  one  range  is  involved,  future 
tests  may  involve  live  assets  at  more  than  one  range  or  even  at  off-range  locations. 
Current  policy  would  prevent  centralized  asset  control,  which  could  result  in 
ineffective  test  control. 

Currently,  only  architectures  involving  a  single  shooter  engaging  a  single  target  with  a 
single  PGM  have  been  investigated  by  JADS  for  PGM  T&E  applications.  In 
principle  it  should  be  possible  to  link  multiple  shooters,  targets,  and  weapons. 
However,  there  will  be  latency  and  synchronization  issues  to  be  addressed,  along  with 
implementation  challenges  yet  to  be  discovered  (i.e.,  the  “unknown  unknowns”). 
Also,  if  ADS  is  used  to  augment  scenarios  involving  live  players  with  simulated 
entities,  the  augmented  scenario  will  only  be  experienced  by  the  simulated  entities 
and  not  by  the  live  players,  due  to  the  general  inability  to  provide  live  platforms  with 
useful  information  on  simulated  entities. 

There  are  concerns  which  must  be  addressed  for  the  proper  implementation  of  ADS.  These 
include  the  following: 

-  The  cost  of  implementing  ADS.  Most  existing  facilities  were  not  designed  to  be  linked 
and  require  some  modification,  along  with  appropriate  interfaces,  before  linking  is 
possible.  This  cost  can  be  significant  (e.g.,  about  $2  million  for  LFP),  but  can  be 
amortized  over  all  phases  of  a  PGM  development  and  testing  program. 

The  launcher/PGM  interface.  Those  PGM  systems  which  use  launcher  support  require 
special  purpose  interfaces  for  launcher-generated  messages  passed  between  the  launcher 
and  the  PGM  HWIL  laboratory.  Interface  requirements  must  be  determined  early  during 
ADS  implementation  planning  to  permit  testing  schedules  to  be  met. 

-  Performing  PGM  integration  testing  for  advanced  launch  platforms.  Some  advanced 
fighters  are  being  designed  to  carry  missiles  internally  and  do  not  have  external  telemetry 
antennas.  This  configuration  will  make  it  difficult  to  implement  the  live  shooter-target 
ADS  architecture  using  the  type  of  instrumentation  employed  during  the  LFP  testing.* It 
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will  likely  be  necessary  to  specially  modify  one  of  these  launch  platforms  to  collect  the 
necessary  telemetry  data  and  permit  its  transmission  to  ground  receivers. 

-  Determining  which  standards  to  use  for  data  exchanges  among  the  ADS  nodes.  DIS  PDU 
standards  do  not  support  all  required  data  exchanges  and  are  not  always  the  best  choice. 
This  concern  is  best  addressed  by  tailoring  data  protocols  for  each  specific  application 
and  implementing  the  principles  of  HLA. 

Use  of  dedicated  links.  The  use  of  dedicated  leased  lines  may  be  justified,  rather  than 
using  existing  networks  such  as  DSI,  due  to  latency  and  reliability  requirements  and 
scheduling  constraints  [Refs.  10,  11].  Each  linking  application  must  evaluate  its 
requirements  and  justify  the  use  of  commercial  links  when  appropriate. 

-  Programmatic  concerns. 

--  Management  and  scheduling  are  more  complex  and  challenging  with  distributed  tests. 
—  ADS  implementation  programs  should  consider  forming  a  separate  or  special  program 
office  with  dedicated  personnel  to  handle  the  financial,  contractual,  and  mission 
support  issues. 

--  A  formal  agreement  or  memorandum  of  understanding  should  be  established  between 
the  testing  organization  and  the  providers  of  test  support  (e.g.,  test  ranges  or  HWEL 
facilities). 

--  Security  can  be  a  major  concern  when  linking  facilities  with  different  security 
requirements. 

A  number  of  methodologies  apply  for  ADS  implementation.  These  are  to  be  developed  by  the 
JADS  JTF  at  some  later  date  and  are  simply  listed  here: 

-  Test  planning  (including  cost  benefit  analysis). 

ADS  architecture  and  instrumentation  design. 

-  Verification  and  validation  (V&V)  of  the  ADS  configuration. 

Integration  of  linked  assets. 

-  Test  control. 

4.0  ADS  Applicability  to  T&E  of  Specific  PGM  types 
4.1  ADS  Applicability  Assessment  Approach 

The  results  from  the  SIT  LSP  and  LFP  testing  are  generalized  by  considering  other  possible 
representations  for  the  shooter,  target,  and  PGM:  The  shooter  and  target  can  be  represented  by 
either  DSMs,  HWIL  laboratories,  or  live  platforms  and  the  PGM  can  be  represented  by  either 
DSMs  or  HWIL  laboratories.  Linking  requirements  and  constraints  learned  from  the  SIT  testing 
are  applied  to  other  ADS  architectures  where  appropriate. 

The  following  principles  are  used  to  determine  the  best  ADS  architecture  for  a  given  test: 

-  The  choice  of  representation  for  each  player  depends  on  (1)  the  test  objectives,  (2)  the 
availability  of  adequate  DSMs  and  HWIL  laboratories  for  simulated  players  and 
instrumentation  for  live  platforms,  and  (3)  the  details  of  the  test  scenarios. 

—  The  advantages  of  DSMs  include  low  cost,  good  availability,  and  high  reproducibility 
of  results.  The  disadvantages  include  the  difficulty  in  adequately  modeling  human-in- 
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the-loop  decisions  and  actions  (DSMs  usually  only  model  scripted  shooter/target 
profiles)  and  their  lower  fidelity  compared  to  HWIL  laboratories  or  live  platforms. 

-  The  advantages  of  HWIL  laboratories  include  moderate  cost,  availability,  and 
reproducibility  of  profiles  and  the  use  of  human/hardware-in-the-loop.  The  replay 
ability  of  HWIL  laboratories  makes  them  especially  advantageous  for  parametric 
studies.  The  disadvantages  include  lower  fidelity  than  live  platforms,  especially  for 
environmental  effects. 

-  The  advantages  of  live  platforms  include  the  use  of  real  shooter  and  target  platforms 
m  real  environments.  Realistic  testing  of  shooter  data  link  support  to  the  PGM  and 
tactics  development  are  best  done  with  live  shooter/target  platforms.  The 
disadvantages  include  high  cost,  low  reproducibility  of  profiles,  general  lack  of  PGM 
feedback  to  the  live  platforms,  and  limited  TSPI  accuracy  for  dynamic  platforms 
without  significant  processing  latency. 

Some  combinations  of  player  representation  may  not  be  possible. 

The  linking  of  a  live  shooter  (or  target)  with  a  simulated  target  (or  shooter)  may  not 
be  possible  when  the  shooter  and  target  are  both  manned  and  interact  with  each  other 
because  the  means  of  providing  the  live  platform  operator  with  realistic 
information/cues  on  the  simulated  entity  do  not  exist,  in  many  cases.  However,  if  the 
target  does  not  react  to  the  shooter,  a  live  target  executing  a  scripted  nonreactive 
profile  could  be  linked  to  a  manned  shooter  laboratory  [Ref.  34],  Such  a  live  target/ 
simulated  shooter  combination  could  be  useful  in  augmenting  live  scenarios  by 

adding  an  additional  simulated  shooter  to  a  few-on-few  scenario  in  which  the  other 
players  are  live. 

-  DSMs  can  be  interactively  linked  with  HWIL  laboratories  or  live  platforms  only  if  the 
DSMs  run  in  real  time.  If  the  player  represented  by  the  DSM  does  not  interact  with 
other  players  (e.g.,  a  missile  fired  at  a  nonreactive  target),  then  the  DSM  does  not 
have  to  run  in  real  time. 

High-fidelity  results  require  synchronization  between  the  players. 

If  the  players  must  interact  with  each  other  in  real  time  (closed-loop  case),  this 
synchronization  is  best  achieved  by  using  second  order  position  dead  reckoning 
(which  uses  the  entity’s  velocity  and  acceleration  to  predict  its  position)  and  second 
order  orientation  dead  reckoning  (which  uses  the  entity’s  attitude  rates  and  angular 
acceleration  to  predict  its  orientation).  The  dead  reckoning  is  performed  at  the  frame 
rate  of  the  receiving  entity  based  on  absolute  time  and  uses  extrapolation  to  correct 
for  latency  and  latency  variations. 

-  If  the  players  do  not  interact  in  real  time  (open-loop  case),  this  synchronization  is  best 
achieved  by  buffering  the  data  exchanged  and  interpolating  it  at  the  frame  rate  of  the 
receiving  entity  based  on  absolute  time  stamps.  Since  interpolation  is  more  accurate 
than  extrapolation,  this  synchronization  technique  results  in  more  accurate  data  being 
provided  to  the  receiving  entity.  The  buffering  results  in  additional  latency,  but  does 
not  affect  results  for  open-loop  cases  (e.g.,  scripted  target  profile). 

Closed-loop  interactions  require  acceptable  latencies,  dead  reckoning  corrections,  and  the 

means  for  feedback  between  players. 

The  amount  of  acceptable  latency  for  a  closed-loop  interaction  depends  on  the  nature 
of  the  reactions  involved. 
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—  For  maneuver  reactions,  the  amount  of  acceptable  latency  depends  on  the  rates  of 
change  of  the  translational  and  rotational  accelerations  of  the  entities  involved 
(since  second  order  position  and  orientation  dead  reckoning  is  used)  and  the 
allowable  position  and  orientation  errors.  For  highly  dynamic  missile-fighter 
aircraft  interactions,  the  allowable  latency  is  on  the  order  of  100  milliseconds. 

—  For  nonmaneuver,  discrete  event  reactions  (e.g.,  flare,  smoke,  or  chaff  deployment 
or  ECM  initiation),  the  amount  of  acceptable  latency  is  typically  about  50 
milliseconds  [Ref.  18],  independent  of  the  motion  of  the  entities  involved. 

--  The  requirement  for  closed-loop  interactions  may  constrain  the  types  of  player 
representations  which  can  be  used. 

—  If  the  shooter  and  target  react  to  each  other  prior  to  PGM  launch  (a  typical 
scenario),  then  typically  the  representation  for  both  must  be  either  manned  HWIL 
laboratories  or  live  platforms. 

—  If  the  target  reacts  to  the  PGM,  then  typically  the  representation  for  the  target  must 
be  a  manned  HWIL  laboratory  (since  feedback  to  a  live  target  platform  is 
generally  not  possible). 

-  The  use  of  DSMs  may  simplify  the  ADS  architecture  if  the  DSM  can  be  hosted  on  the 
computer  controlling  another  simulated  entity.  In  this  case,  linking  between  the  DSM  and 
its  host  is  not  necessary.  For  example,  many  manned  aircraft  simulators  have  embedded 
air-to-air  missile  DSMs. 

4.2  ADS  Applicability  Assessment  Results 

The  general  PGM  classes  are  examined  to  determine  which  linking  configurations  can  support 
testing  of  each  class.  For  each  class,  the  basic  engagement  of  one  shooter  engaging  one  target 
with  one  PGM  is  considered.  (The  basic  engagement  can  be  augmented  using  ADS,  as 
previously  discussed.)  The  general  PGM  classes  are  as  follows: 

-  Air-to-air  missile  (AAM). 

Surface-to-air  missile  (SAM). 

-  Air-to-ground  munitions  (AGM)  and  air-to-subsurface  munitions. 

Surface-to-surface  and  subsurface  munitions. 

4.2.1  Air-to-Air  Missile  Applications 

AAM  scenarios  involve  a  highly  dynamic  shooter  and  target.  This  PGM  class  was  directly 
evaluated  during  the  LSP  and  LFP  testing.  Current  examples  of  this  class  are  AIM-9X  and  AIM- 
120C.  General  considerations  for  applying  ADS  to  this  PGM  class  are  given  in  Section  4.1. 
Special  considerations  are  as  follows: 

-  If  the  target  interacts  with  the  missile  (reactive  target)  in  a  closed-loop  fashion,  the 
shooter  and  target  should  be  represented  by  manned  HWIL  laboratories  and  the  missile  by 
either  an  HWIL  laboratory  or  a  DSM. 

-  If  the  shooter  HWIL  laboratory  is  linked  to  a  missile  HWIL  laboratory,  a  special 
purpose  interface  will  be  required  to  pass  initialization,  launch,  and  targeting 
messages  to  the  missile. 

—  Low  latencies  are  required  (<100  milliseconds)  with  second  order  dead  reckoning. 
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The  nature  of  the  cues  used  by  the  target  pilot,  to  react  to  the  missile  may  dictate  the 
type  of  simulator  needed  for  the  target.  For  example,  if  visual  cues  are  needed,  a 
domed  aircraft  simulator  with  a  high-fidelity,  out-the-window  display  may  be  required 
(see  Appendix  A).  However,  if  the  pilot  primarily  relies  on  a  missile  warning  system 
without  visual  cues,  a  domed  simulator  would  not  be  needed. 

A  live  shooter-target  ADS  architecture  (live  shooter  and  live  target  linked  to  HWIL  or 

DSM  missile)  is  best  used  to  evaluate  aircraft  engagement  tactics  and  data  link  support  to 
the  missile. 

If  the  live  shooter  is  linked  to  a  missile  HWIL  laboratory,  a  special  purpose  interface 
will  be  required  to  pass  initialization,  launch,  and  targeting  messages  to  the  missile. 
Evaluation  of  the  data  link  message  accuracy  may  require  highly  accurate  shooter  and 
target  TSPI  data  (TSPI  accuracy  should  be  about  a  factor  of  ten  better  than  data  link 
message  accuracy),  and  required  processing  times  may  prevent  this  architecture  from 
running  in  real  time. 

-  If  shooter  support  is  provided  via  fire  control  radar  (FCR)  illumination  of  the  target 
for  semi-active  RF  guidance,  there  is  no  real  advantage  to  using  the  live  shooter-target 
configuration  for  evaluating  the  quality  of  support.  This  is  because  both  the  shooter 
FCR  return  from  target  and  the  direct  shooter  reference  signal  received  by  the  missile 

cannot  be  directly  measured  in  the  live  environment  and  must  be  simulated  at  the 
missile  node. 

4.2.2  Surface-to-Air  Missile  Applications 

SAM  scenarios  involve  a  stationary  or  slow  moving  surface  launcher  and  a  highly  dynamic, 
reactive  target.  This  ADS  application  has  been  investigated  in  previous  studies  [Ref  12] 
General  considerations  for  applying  ADS  to  this  PGM  class  are  given  in  Section  4.1.  Special 
considerations  are  as  follows: 

The  shooter  (launcher)  may  or  may  not  have  to  be  represented  as  a  separate  entity. 

The  launcher  and  its  radar  are  often  part  of  the  SAM  simulation.  In  this  case,  linking 
between  the  shooter  and  SAM  is  not  required.  & 

-  If  the  launcher  is  mobile,  it  may  be  desirable  to  use  a  manned  HWIL  laboratory  or  live 
platform  for  the  shooter,  depending  on  the  nature  of  the  scenario  and  the  test 
objectives  (e.g.,  if  the  shooter  maneuvers  in  response  to  the  target  before  launching 
the  SAM  or  if  a  human  decision  is  needed  to  launch  the  SAM). 

A  live  shooter  can  be  linked  to  a  simulated  target  in  this  case  because  the  shooter  only 
relies  on  radar  detection,  and  this  can  be  simulated. 

If  the  target  reacts  to  the  missile,  the  target  should  be  represented  by  a  manned  HWIL 
laboratory  and  the  missile  by  either  an  HWIL  laboratory  or  a  DSM. 

--  Low  latencies  are  required  (<  1 00  milliseconds)  with  second  order  dead  reckoning. 

-  As  for  the  AAM  application,  the  nature  of  the  cues  used  by  the  target  pilot  to  react  to 
the  missile  may  dictate  the  type  of  simulator  needed  for  the  target. 

-  If  the  target  reacts  to  the  shooter  radar  by  employing  ECM,  but  does  not  react  to  the 
missile,  either  a  linked  laboratory  or  a  live  shooter-target  ADS  architecture  can  be  used. 
(The  ECM  can  be  designed  to  affect  both  the  shooter  radar  and  the  SAM  seeker,  but  its 
employment  is  a  reaction  to  the  shooter  radar  only.) 
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—  The  linked  laboratory  architecture  would  have  all  three  players  represented  by  HWIL 
laboratories  (the  shooter  and  SAM  might  be  represented  by  the  same  HWIL 
laboratory).  The  target  would  determine  when  the  ECM  begins,  but  the  actual  ECM 
would  be  applied  in  the  shooter/PGM  HWIL  laboratory. 

--  The  live  shooter-target  ADS  architecture  would  use  a  live  shooter  radar  tracking  a  live 
target  aircraft  with  an  ECM  pod.  The  live  players  would  be  linked  to  a  SAM  HWIL 
laboratory  for  the  missile  flyout,  and  ECM  would  also  be  applied  in  the  laboratory  if 
the  ECM  is  designed  to  affect  the  SAM,  as  well  as  the  shooter  radar. 

-  A  live  shooter-target  ADS  architecture  is  best  used  to  evaluate  realistic  shooter  radar 

performance  against  a  real  target  which  may  be  employing  ECM  against  the  shooter. 

—  Evaluation  of  the  launcher  radar  tracking  accuracy  requires  highly  accurate  target 
TSPI  data,  and  required  processing  times  may  prevent  real-time  operation. 

4.2.3  Air-to-Ground  Munitions  and  Air-to-Subsurface  Munitions  Applications 

These  scenarios  involve  a  dynamic  shooter  and  either  a  stationary  or  slow  moving  ground  or 
subsurface  target.  Some  types  of  these  munitions  require  shooter  support,  such  as  laser 
illumination  of  the  target  or  data  link  messages.  Also,  the  support  can  be  from  a  platform  other 
than  the  shooter  (third  party  support).  Current  examples  of  AGMs  are  JDAM,  HARM,  JSOW, 
JASSM,  SLAM-ER.  General  considerations  for  applying  ADS  to  this  PGM  class  are  given  in 
Section  4. 1 .  Special  considerations  are  as  follows: 

-  ADS  implementation  would  be  well-suited  to  the  evaluation  of  AGM  attacks  on  targets 

employing  reactive  CM. 

—  The  shooter  would  best  be  represented  by  a  manned  HWIL  laboratory  and  the  AGM 
by  either  a  DSM  or  an  HWIL  laboratory. 

-  The  target  representation  required  would  depend  on  whether  the  target  is  fixed  or 
mobile  and  whether  maneuvering  is  part  of  its  reaction.  (In  general,  these  targets  are 
slow  moving  so  that  maneuvering  would  not  be  an  effective  CM.)  For  targets  which 
do  not  maneuver,  but  only  employ  CM  techniques  such  as  flares,  smoke,  ECM,  etc., 
the  target  representation  could  be  simplified  to  only  a  shooter/launch  detection  and  a 
CM  deployment  function. 

-  CM  designed  to  counter  the  AGM  would  be  added  to  the  seeker  scene  in  the  AGM 
DSM  or  HWIL  laboratory  when  employed  by  the  target.  Latency  requirements  would 
depend  on  the  nature  of  the  CM  and  scenario  details  and  must  be  analyzed  on  a  case- 
by-case  basis. 

-  A  live  shooter-target  ADS  architecture  is  best  used  to  evaluate  realistic  support  of  the 

AGM  by  laser  illumination  of  the  target. 

—  The  laser  hit  spot  on  the  target  and  the  reflected  laser  energy  would  be  measured  by  a 
detector.  The  measurement  would  be  used  to  dynamically  scale  and  position  the  laser 
energy  presented  to  the  AGM  seeker  in  an  HWIL  laboratory. 

-  The  laser  energy  detector  could  be  located  on  the  shooter,  but  would  generally  be  at 
an  independent  (possibly  fixed)  location.  When  the  detector  is  at  an  independent 
location,  a  separate  node  would  be  added  to  the  ADS  architecture. 

-  If  the  laser  illumination  is  provided  by  a  third  party,  entity  state  or  other  data  may  be 
required  for  this  additional  player.  In  this  case,  another  node  would  be  added  to  the 
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ADS  architecture  for  the  third  party.  (However,  the  AGM  and  shooter  may  not  need 
to  “know”  anything  about  the  third  party,  but  only  the  intensity  of  the  reflected  laser 
energy.  In  that  case,  no  data  may  have  to  be  passed  directly  from  the  third  party.) 

-  A  live  shooter-target  ADS  configuration  could  also  be  appropriate  if  target  CM  are 
directed  against  the  shooter  rather  than  the  AGM. 

4.2.4  Surface-to-Surface  and  Subsurface  Applications 

These  scenarios  involve  a  slow  or  stationary  shooter  and  target.  As  for  AGM,  some  types  of 
these  munitions  require  shooter  support.  The  surface-to-surface  ADS  application  is  currently 
being  investigated  for  FOTT  testing  [Refs.  28,  35],  and  the  subsurface  application  is  being 
investigated  for  torpedo  testing  and  training  [Refs.  29,  30].  General  considerations  for  applying 
ADS  to  this  PGM  class  are  given  in  Section  4.1.  Special  considerations  are  as  follows: 

Most  of  the  special  considerations  for  AGM  and  air-to-subsurface  munitions  applications 
also  apply  to  these  PGM  classes. 

The  live  shooter-target  ADS  architecture  cannot  be  used  to  test  wire-guided  munitions,  in 
general,  since  feedback  between  the  shooter  and  the  PGM  is  needed. 

-  Wire-guided  munitions  are  best  tested  using  linked  laboratories  with  the  shooter  and 
PGM  represented  by  HWIL  laboratories  and  the  target  by  either  a  DSM  or  HWIL 
laboratory.  This  is  the  architecture  currently  being  developed  for  FOTT  testing  at 
Redstone  Arsenal,  Alabama,  [Ref.  35]  although  future  planned  enhancements  will  slave  a 
live  shooter  (soldier)  on  a  test  range  to  the  shooter  HWIL  laboratory. 

The  Synthetic  Environment  Tactical  Integration  Virtual  Torpedo  Project  (SETI  VTP)  at 
the  Naval  Undersea  Warfare  Center  (NUWC)  in  Newport,  Rhode  Island,  is  developing  an 
HLA  federation  to  link  live  submarines  to  a  high-fidelity  torpedo  HWIL  facility  [Refs.  29, 


5.0  Summary 

The  findings  from  the  two  phases  of  SIT  testing  were  extrapolated  to  PGM  classes  other  than 
AAM,  and  it  was  determined  that  various  ADS  architectures  appear  to  have  utility  for  supporting 
PGM  T&E  in  general  and  that  there  are  benefits  to  using  ADS  when  it  is  appropriate.  The 
shooter  and  target  can  be  represented  by  live  platforms,  manned  HWEL  laboratories,  or  DSMs, 
and  the  PGM  can  be  represented  by  either  HWIL  laboratories  or  DSMs.  The  choice  of  player 
representation  depends  on  (1)  the  test  objective,  (2)  the  availability  of  adequate  DSMs  and  HWIL 
laboratories  for  simulated  players  and  instrumentation  for  live  platforms,  and  (3)  the  details  of 
the  test  scenarios  (such  as  whether  the  target  reacts  to  the  PGM  or  not). 

The  assessment  gave  general  guidelines  for  ADS  implementation  for  various  classes  of  PGM. 
Detailed  requirements  for  linking  specific  PGM  systems  (other  than  the  AIM-9M  and  AIM- 120 
missiles  addressed  by  the  LSP  and  LFP)  were  not  developed.  Also,  some  characteristics  of 
specific  PGM  systems  may  have  been  overlooked  which  could  impact  the  ADS  architecture 
design  and  feasibility  of  implementation.  The  extrapolation  of  SIT  results  to  PGM  classes  other 
than  AAM  was  based  on  informed  conjecture  without  rigorous  analysis  or  supporting  data. 
Applications  were  assumed  to  be  feasible  unless  there  was  evidence  to  the  contrary. 
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6.0  Abbreviations  and  Acronym  List 


A/C  aircraft 

AAM  air-to-air  missile 

AASI  Advanced  Aircraft  Simulation  Interface 

ADS  advanced  distributed  simulation 

AFB  Air  Force  Base 

AFDTC  Air  Force  Developmental  Test  Center 

AGM  air-to-ground  munitions 

AIM  air  intercept  missile 

AMRAAM  Advanced  Medium  Range  Air-to-Air  Missile 

ASC  Aeronautical  Systems  Center  (at  Wright-Patterson  AFB,  OH) 

BMIC  Battle  Management  Interoperability  Center  (control  facility  at  Point  Mugu,  CA) 

BVR  beyond  visual  range 

C4I  command,  control,  communications,  computers,  and  intelligence 

CCF  Central  Control  Facility  (at  Eglin  AFB,  FL) 

CCM  counter-countermeasures 

CM  countermeasures 

COMSEC  communications  security 

coord  coordination 

CSU  channel  service  unit 

DIS 
DISA 
DISN 
DMSO 
DoD 
DRA 
DSI 
DSM 
DSU 
DT&E 

ECCM 
ECM 
EO 
EOA 
ESSM 
ETE 
EW 

FCR  fire  control  radar 

FEDEP  Federation  Development  and  Execution  Process 


electronic  counter-countermeasures 

electronic  countermeasures 

electro-optical 

early  operational  assessment 

Evolved  Sea  Sparrow  Missile 

End-To-End  Test 

electronic  warfare 


distributed  interactive  simulation 

Defense  Information  Systems  Agency 

Defense  Information  Systems  Network 

Defense  Modeling  and  Simulation  Office  (Alexandria,  VA) 

Department  of  Defense 

dead  reckoning  algorithm 

Defense  Simulation  Internet 

digital  simulation  model 

data  service  unit 

developmental  test  and  evaluation 
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FOM 

FOT&E 

FOTT 

FOV 

F-Pole 


GPS 

HARM 

HLA 

HQ 

HWIL 

INS 

IOT&E 

IR 

IRIG 

JADS 

JASSM 

JDAM 

JSF 

JSOW 

JT&E 

JTF 

lab 

LAN 

LFP 

LSP 

MISILAB 

MOA 

msl 

NTP 

NUWC 

OA 

OT 

OT&E 

OTW 

PDU 


federation  object  model 
follow-on  test  and  evaluation 

Follow-On  To  TOW  (Tube-launched  Optically  tracked  Wire-guided) 
field  of  view 

range  between  the  launch  aircraft  and  the  target  aircraft  at  the  time  an  air-to-air 
missile  intercepts  the  target 

global  positioning  system 

High-speed  Anti-Radiation  Missile 
high  level  architecture 
headquarters 
hardware-in-the-loop 

inertial  navigation  system 

initial  operational  test  and  evaluation 

infrared 

Interrange  Instrumentation  Group 

Joint  Advanced  Distributed  Simulation 

Joint  Air-to-Surface  Standoff  Missile 

Joint  Direct  Attack  Munition 

Joint  Strike  Fighter 

Joint  Stand-Off  Weapon 

Joint  Test  and  Evaluation 

Joint  Test  Force 

laboratory 
local  area  network 
Live  Fly  Phase 
Linked  Simulators  Phase 

Missile  Simulation  Laboratory  (AMRAAM  HWIL  lab  at  Eglin  AFB,  FL) 

memorandum  of  agreement 

missile 

network  time  protocol 

Naval  Undersea  Warfare  Center  (Newport,  Rhode  Island) 

operational  assessment 
operational  test 
operational  test  and  evaluation 
out-the-window 

protocol  data  unit 
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PGM  precision  guided  munitions 

POC  point  of  contact 


RDL 

RF 

RTI 

RWR 


rear  data  link 
radio  frequency 
runtime  infrastructure 
radar  warning  receiver 


SAM 
SETI VTP 


sim 

SIMLAB 

SIPRNET 

SIT 

SLAM-ER 

SMS 

SUT 


surface-to-air  missile 

Synthetic  Environment  Tactical  Integration  Virtual  Torpedo  Project  (at  NUWC, 

Newport,  Rhode  Island) 

simulation 

Simulation  Laboratory  (AIM-9  HWIL  lab  at  China  Lake,  CA) 

Secure  Internet  Protocol  Router  Network 
System  Integration  Test 

Standoff  Land  Attack  Missile  -  Expanded  Response 
stores  management  system 
system  under  test 


T&E 

TCAC 

TDP 

tgt 

TM 

TOW 

TSPI 


test  and  evaluation 

Test  Control  and  Analysis  Center 

TSPI  Data  Processor 

target 

telemetry 

Tube-launched  Optically-tracked  Wire-guided 
time-space-position  information 


umb  umbilical 


V&V  verification  and  validation 

VV&A  verification,  validation,  and  accreditation 


WAN  wide  area  network 

WSIC  Weapon  System  Integration  Center  (F-14D  manned  flight  lab.  Point  Mugu,  CA) 

WSSF  Weapon  System  Support  Facility  (F/A-l  8  manned  flight  lab,  China  Lake,  CA) 


UNITS  OF  MEASURE 


°  degree 

g  acceleration  due  to  gravity  =  32  ft/s2  or  9.8  m/s2 

m  meter 

ms  millisecond 
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USE  OF  DOMED  FLIGHT  SIMULATORS  FOR  MISSILE  TESTING 
A.1.0  Introduction 

A  major  limitation  in  open  air  range  live  fire  testing  of  AAMs  or  SAMs  is  the  inability  to 
evaluate  target  reactive  CM  due  to  prohibitions  against  firing  a  live  missile  at  a  manned  target. 
ADS  can  provide  such  a  testing  capability  by  linking  manned  flight  simulators  to  missile  HWL 
laboratories  or  DSMs.  However,  the  simulators  must  be  capable  of  providing  all  essential  visual, 
aural,  and  sensor  cues  to  the  target  pilot.  In  particular,  the  target  pilot  should  receive  visual 
launch  cues  and  cues  from  missile  warning  systems  [Ref.  A-l]. 

This  study  evaluates  the  value  to  AAM  and  SAM  T&E  of  providing  visual  cues  to  the  target  pilot 
by  use  of  a  domed  manned  flight  simulator  and  the  feasibility  of  existing  domed  simulators  to 
provide  realistic  visual  cues  and  to  support  AAM  and  SAM  T&E. 

A.2.0  Value  to  Air-to-Air  Missile  and  Surface-to-Air  Missile  T&E 

The  primary  value  to  AAM  and  SAM  T&E  of  providing  realistic  visual  cues  to  the  target  pilot  is 
that  more  realistic  evaluations  of  the  effectiveness  of  target  CM  tactics  would  be  possible  (or, 
conversely,  the  effectiveness  of  missile  reactions  to  target  CM  tactics  could  be  evaluated). 
Visual  cues  would  be  useful  for  the  following: 

-  Prelaunch  maneuvering  against  the  shooter  (AAMs  only).  The  best  defensive  tactic  for 
the  target  is  to  prevent  the  shooter  from  achieving  a  favorable  launch  geometry.  This  is 
especially  true  for  short  range  AAMs  (e.g.,  AIM-9)  launched  within  visual  range.  A  high- 
fidelity,  out-the-window  (OTW)  display  in  the  target  simulator  would  allow  realistic 
simulation  of  such  maneuvering  for  short  range  visual  engagements.  However,  it  should 
be  noted  that  most  radar-guided  missiles  (e.g.,  AMRAAM  and  Sparrow)  are  normally 
launched  beyond  visual  range  (BVR),  so  that  visual  prelaunch  cues  will  not  be  available 
for  the  target  pilot  in  such  cases. 

Missile  launch  cues.  The  most  reliable  launch  cue  is  nonvisual  and  comes  from  the 
target’s  radar  warning  receiver  (RWR).  For  BVR  shots  there  will  be  no  visual  launch 
cue.  For  AAM  shots  within  visual  range  the  target  will  use  visual  cues  to  maneuver  in 
response  to  the  shooter  up  to  and  through  launch.  For  both  AAM  and  SAM  launches 
within  visual  range  the  target  may  get  a  visual  launch  cue  such  as  a  puff  of  smoke  or  flash 
from  the  missile  or  may  see  the  missile  leave  the  launcher.  These  cues  from  the  missile 
itself  are  unreliable,  as  they  depend  on  environmental  conditions  and  the  launch  occurring 
within  the  target  pilot’s  field  of  view  (FOV). 

-  Cues  for  post-launch  evasive  maneuvering.  The  target  pilot  will  generally  not  get  visual 
cues  from  the  missile  during  its  flyout,  unless  the  launch  is  at  night  or  the  missile  emits 
visible  smoke.  For  a  smokeless  missile  (typical  of  modem  AAMs)  launched  during 
daylight,  the  target  pilot  will  be  unable  to  discern  the  missile  until  its  range  is  so  small 
(and  its  remaining  time  of  flight  is  so  short)  that  human  reaction  times  prevent  the  pilot 
from  responding  in  time.  For  an  AAM  engagement  within  visual  range,  the  target  pilot 
will  continue  to  maneuver  in  response  to  the  shooter  using  OTW  visual  cues,  such  as  the 
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angle  of  the  horizon  and  ground  references,  in  executing  high-acceleration  evasive 
maneuvers.  A  high-fidelity,  OTW  display  would  provide  these  cues. 

The  conclusion  is  that  a  domed  manned  flight  simulator  with  a  high-fidelity,  OTW  display  would 
add  realism  for  pre-  and  post-launch  evasive  maneuvering  and  would  provide  launch  cues  in 
some  (but  not  all)  engagements.  The  presence  of  visual  missile  launch  and  flyout  cues  and  their 
value  in  aiding  the  target  pilot  depends  on  the  type  of  missile,  environmental  conditions,  and 
engagement  scenario  details,  especially  launch  range. 


A.3.0  Domed  Simulator  Requirements 

In  order  for  a  domed  manned  flight  simulator  to  be  able  to  support  AAM  and  SAM  T&E  it  must 
meet  requirements  for  realism  of  the  visual  display  and  allowable  latency  for  real-time  linking. 

A.3.1  Visual  Display  Requirements 

The  OTW  display  provided  to  the  target  pilot  must  provide  realistic  cues  for  the  missile  launch 
and  for  effective  evasive  maneuvering.  The  suitability  of  displays  for  these  purposes  is  best 
judged  by  experienced  fighter  pilots.  The  relevant  display  qualities  were  defined  by  pilots  in  an 
Aeronautical  Systems  Center  (ASC)  technical  report  [Ref.  A-2]  and  are  summarized  as  follows: 

-  Full  FOV.  A  full  FOV,  including  the  periphery,  is  needed  by  the  pilot  to  perform  the 
visual  tasks  encountered  during  combat.  Anything  less  than  full  FOV  viewing  limits 
combat  tasks  that  can  be  performed  in  a  flight  simulator.  The  display  must  respond  to 

pilot  head  and  eye  movement  without  requiring  exaggerated  and  unrealistic  head 
movement. 

Adequate  resolution  and  detail.  The  resolution  of  various  scene  features  needs  to  be 
sufficient  to  provide  essential  visual  cues  to  the  pilot. 

There  must  be  adequate  resolution  in  the  periphery. 

The  visual  display  of  missiles  and  aircraft  must  be  sufficiently  detailed. 

Ground  features  must  be  presented  with  adequate  detail  (resolution)  and  texture  to 
allow  the  pilot  to  determine  altitude  visually. 

Realistic  contrast.  The  visual  display  of  threat  aircraft  must  be  realistic  in  contrast  against 
the  background  to  properly  aid  in  pilot  detection  and  tracking. 

-  Realistic  entity  presentation.  The  visual  display  of  missiles  and  aircraft  must  be  realistic 
in  color,  relative  size,  detail,  and  any  relevant  emissions  (e.g.,  smoke  or  fire). 

Minimal  latency.  The  display  must  not  have  a  perceptible  lag  (i.e.,  latency). 

A.3.2  Latency  Requirements 

The  processing  of  data  at  ADS  nodes  and  the  transmission  of  data  among  nodes  results  in  delay 
or  latency  in  the  receipt  of  data.  Latency,  if  not  properly  compensated  for,  can  prevent  an 
accurate  interaction  between  simulation  entities.  In  particular,  latency  and  its  variance 
compromise  spatial  correlation. 
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In  a  study  of  latency  effects  on  aircraft  simulations,  a  testbed  was  constructed  using  DIS- 
compatible  F-15  simulations  [Ref.  A-3],  DIS  PDU  output  of  an  F-15  was  passed  to  a  network, 
subjected  to  a  variable  delay,  and  then  passed  back  to  the  network  interface  just  as  though  it  had 
been  received  from  the  network.  The  position  and  orientation  of  the  entities  received  from  the 
network  were  compared  to  that  of  the  internal  models  and  the  average  and  maximum  differences 
were  recorded.  Dead  reckoning  algorithms  (DRAs)  were  used  to  reduce  the  position  and 
orientation  differences,  and  the  smallest  differences  were  for  DRAs  which  used  second  order 
*position  dead  reckoning  (which  uses  the  entity’s  velocity  and  acceleration  to  predict  its  position) 
and  first  order  orientation  dead  reckoning  (which  uses  the  entity’s  attitude  rates  to  predict  its 
orientation).  Resulting  errors  when  the  aircraft  simulation  was  executing  highly  dynamic 
maneuvers  (with  instantaneous  accelerations  up  to  9  g)  are  given  in  Table  A-l  for  various 
network  delays. 


Table  A-l.  Aircraft  Simulator  Errors  for  Highly  Dynamic  Maneuvers  Versus  Network 
Delay  (second  order  position,  first  order  orientation  DRA  used)  [from  Ref.  A-3]. 


Network 

Delay 

(ms) 

Average 
Position 
Error  (m) 

Maximum 
Position 
Error  (m) 

Average 
Orientation 
Error  (°) 

Maximum 
Orientation 
Error  (°) 

0 

0.43 

2.4 

0.89 

4.57 

100 

0.57 

3.79 

1.5 

8.26 

200 

0.87 

6.03 

2.55 

13.5 

300 

1.46 

8.71 

4.23 

21.03 

400 

2.16 

10.25 

5.99 

29.21 

500 

3.13 

13.56 

8.01 

38.03 

The  results  in  the  table  show  that  the  total  latency  must  be  no  larger  than  about  200  ms  if  average 
position  errors  are  to  be  less  than  1  m  and  no  larger  than  about  100  ms  if  average  orientation 
errors  are  to  be  about  1  °.  However,  position  errors  up  to  almost  4  m  and  orientation  errors  up  to 
more  than  8°  can  occur  for  a  latency  of  100  ms.  Reducing  the  maximum  orientation  error  to  1° 
or  less  will  require  the  use  of  second  order  orientation  dead  reckoning  (which  uses  the  entity’s 
attitude  rates  and  angular  acceleration  to  predict  its  orientation).  These  results  apply  to  two 
aircraft  maneuvering  against  each  other. 

For  the  situation  of  a  missile  interacting  with  a  target  fighter,  the  latency  requirement  for  position 
accuracy  can  be  analyzed  based  on  the  use  of  a  second  order  position  DRA  and  the  maximum 
missile  acceleration.  From  LFP  data,  the  largest  missile  acceleration  was  about  20  g.  Then,  the 
total  latency  must  be  no  larger  than  100  ms  if  the  missile  position  error  is  to  be  1  m  or  less  [Ref. 
A -4], 

The  criterion  for  the  allowable  position  error  is  that  the  maximum  error  must  be  a  fraction  (10% 
to  50%)  of  the  missile’s  lethal  radius,  so  that  the  missile  and  target  do  not  disagree  on  whether  or 
not  the  target  was  killed.  Based  on  this  criterion,  the  latency  must  be  less  than  100  ms  and 
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second  order  position  and  orientation  dead  reckoning  must  be  used.  However,  certain  event 
information  (e.g.,  missile  launch)  cannot  be  predicted,  and  latency  compensation  cannot  be 
provided  for  these  events.  For  such  events,  there  are  absolute  latency  requirements  which  depend 
on  the  scenario  details,  such  as  how  the  target  processes  launch  notifications.  A  rule  of  thumb  is 
that  allowable  latencies  for  discrete  events  are  about  50-100  ms  [Ref.  A-4]. 

The  conclusion  is  that  end-to-end  latencies  between  a  missile  simulation  (either  an  HWIL 
laboratory  or  a  DSM)  and  a  manned  domed  aircraft  simulator  must  be  50-100  ms. 

A.4.0  Domed  Simulator  Evaluation 

Current  manned  domed  aircraft  simulators  are  now  evaluated  against  the  requirements  identified 
in  Section  A.3. 

A.4.1  Visual  Display  Requirements  Evaluation 

The  conclusion  of  the  ASC  technical  report  [Ref.  A-2]  was  that  current  systems  do  not  provide 
adequate  resolution,  contrast,  and  brightness  to  allow  realistic  air-to-air  engagements.  Some  of 
the  specific  deficiencies  are  as  follows: 

A  number  of  tasks  could  not  be  adequately  performed.  These  include  visual  lookout, 
detect  visual  threats,  single  ship  threat  reactions,  and  low  altitude  intercept. 

-  Most  simulators  did  not  have  realistic  threat  presentations,  including: 

Threats  were  too  easy  to  see  due  to  their  exaggeration  and/or  bright  false  color. 

--  Threats  did  not  have  proper  contrast  to  the  background. 

Threats  lacked  sufficient  detail  to  allow  judging  range,  closure  rate,  and  aspect  angle. 

-  The  ground  presentation  was  not  realistic,  so  that  the  pilot’s  altitude  could  not  be  judged 

based  on  visual  cues.  & 

Some  simulators  did  not  respond  to  eye  movement  and  required  exaggerated  head 
movement  to  view  scene  details. 

-  The  helmet-mounted  display  evaluated  did  not  allow  pilots  to  easily  cross-check  cockpit 
instruments  and  restricted  the  pilots’  head  movements  for  checking  aft  areas. 

The  pilots  reported  noticeable  lags  (latency)  in  the  visual  scene  response  to  head 
movements. 

All  visual  systems  had  at  least  one  of  these  deficiencies,  so  that  none  were  judged  to  be  adequate. 

The  ASC  report  included  recommendations  for  improving  the  visual  systems  in  future  aircraft 
simulators. 

A.4.2  Latency  Requirements  Evaluation 

A  number  of  measurements  of  the  latencies  of  linked  domed  manned  aircraft  simulators  have 
been  performed.  When  existing  simulators  are  linked  with  no  attempt  to  reduce  processing  times 
or  improve  interfaces,  latencies  as  high  as  700  ms  have  been  measured  between  the  initiation  of  a 
maneuver  at  one  simulator  and  the  observation  of  the  maneuver  at  another  simulator  [Ref.  A-5], 
By  modifying  the  image  generation  system  and  upgrading  interfaces,  Wright  Laboratory,  Wright- 
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Patterson  Air  Force  Base,  Ohio,  was  able  to  reduce  the  latency  of  the  image  response  to  a 
maneuver  from  about  500  ms  to  about  225-250  ms  for  an  unlinked  domed  simulator  [Ref.  A-6]. 

Even  with  special  modifications,  the  latency  of  a  linked  domed  aircraft  simulator  significantly 
exceeded  the  requirement  of  50-100  ms. 

However,  if  a  lower  fidelity  DSM  representation  is  used  for  the  missile,  rather  than  an  HWIL 
laboratory,  it  should  be  possible  to  reduce  the  latency  between  a  domed  aircraft  simulator 
(representing  the  target)  and  the  missile  DSM  to  the  required  range,  especially  if  the  missile 
DSM  is  hosted  on  the  same  computer  that  controls  the  aircraft  simulator.  The  target  domed 
simulator  could  be  linked  to  a  shooter  man-in-the-loop  simulator  (which  could  be  a  domed 
simulator  for  AAM  testing)  for  realistic  pre-  and  post-launch  maneuvering  against  the  shooter 
and  shooter  initiation  of  the  launch.  The  shooter-target  link  would  have  relaxed  latency 
requirements  (compared  to  the  50-100  ms  value  between  the  target  and  missile)  which  should  be 
achievable  with  existing  simulators. 

A.5.0  Conclusions  and  Recommendations 

Current  domed  manned  aircraft  simulators  are  inadequate  for  supporting  AAM  and  SAM  T&E  in 
which  closed-loop  interactions  between  the  missile  and  target  aircraft  are  investigated.  The 
visual  displays  do  not  provide  realistic  cues,  and  the  latencies  of  domed  simulators  are  too  large 
when  linked  to  a  missile  HWIL  laboratory. 

The  use  of  domed  simulators  with  suitable  performance  attributes  would  provide  added  value  to 
the  investigation  of  closed-loop  interactions.  Hence,  it  is  recommended  that  progress  in  future 
domed  simulators  be  monitored  for  potential  implementation  in  ADS-based  testing  involving 
missile  HWIL  laboratory. 

In  the  meantime,  there  are  two  lower  fidelity  options  for  evaluating  closed-loop  target-missile 
interactions: 

-  The  use  of  a  DSM  for  the  missile  linked  to  a  domed  simulator  for  the  target. 

-  The  use  of  manned  aircraft  laboratories  which  do  not  have  high-fidelity  visual  displays 
(nondomed  laboratories)  linked  to  a  missile  HWIL  laboratory.  In  this  case,  the  cue  for  the 
target  pilot  is  provided  by  a  missile  warning  system  or  other  nonvisual  means.  The 
latencies  for  nondomed  aircraft  laboratories  are  significantly  smaller  than  for  domed 
simulators;  during  LSP  testing,  the  average  latency  between  the  WSIC  target  laboratory 
and  the  SIMLAB  AIM-9M  HWIL  simulation  was  70  ms  [Ref.  A-7]. 

Either  low-fidelity  option  could  permit  the  development  of  missile  warning  system  requirements 
(e.g.,  how  much  warning  time  is  required  to  increase  target  survivability?)  or  evaluation  of  the 
effectiveness  of  missile  responses  (i.e.,  counter-countermeasures)  to  reactive  target  CM  cued  by  a 
missile  warning  system. 
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APPENDIX  B 

COST  BENEFIT  ANALYSIS  FOR  ADS-BASED  TESTING 


A  study  performed  by  the  JADS  JTF  System  Integration  Test  Team 


37 


38 


COST  BENEFIT  ANALYSIS  FOR  ADS-BASED  TESTING 


B.1.0  Introduction 

The  SIT  LFP  and  LSP  ADS  architectures  provided  valid  results  for  air-to-air  missile  T&E  and 
have  been  judged  to  have  utility  for  several  types  of  missile  performance  and  launcher  integration 
evaluations.  Because  of  the  high  cost  of  expending  missiles  and  destroying  target  drones  in  live 
fire  tests,  a  high-fidelity,  non-destructive  ADS-based  test  of  the  missile  has  the  potential  to  save 
significant  money,  especially  if  ADS-based  testing  is  utilized  over  several  phases  of  test. 

This  appendix  lays  out  a  general  methodology  for  determining  the  cost  effectiveness  of 
implementing  ADS-based  testing  in  a  PGM  test  program  and  provides  a  cost  savings  analysis 
example.  Two  approaches  to  cost  effectiveness  are  addressed:  (1)  a  cost  savings  approach  and 
(2)  a  more  effective  test  approach.  For  the  first  approach  total  testing  costs  are  reduced  by 
replacing  a  limited  number  of  live  fire  tests  with  ADS-based  tests.  In  the  second  approach,  ADS- 
based  tests  are  added  without  eliminating  any  live  fire  tests,  so  that  the  total  testing  costs  are 
higher,  but  with  some  compensating  qualitative  benefits. 

B.2.0  General  Methodology 

The  general  cost-effectiveness  methodology  focuses  on  qualitative  benefits  as  well  as 
quantitative  cost  factors.  The  outline  of  the  methodology  is  as  follows: 

-  Analyze  the  appropriate  use  of  ADS  in  the  total  acquisition  and  testing  program. 

—  Estimate  the  potential  benefits  of  implementing  ADS-based  testing. 

-  Determine  any  probable  testing  constraints  which  might  limit  ADS  implementation. 

-  Estimate  costs  with  and  without  ADS  implementation. 

—  If  ADS-based  testing  results  in  lower  testing  costs,  quantify  cost  savings  and  assess 
benefits  and  deficits  of  ADS  use.  (Slight  deficits  may  be  outweighed  by  significant 
cost  savings.) 

—  If  ADS-based  testing  costs  about  the  same,  determine  the  benefits  or  deficits  of  using 
ADS. 

-  If  ADS-based  testing  results  in  higher  total  testing  costs,  determine  if  the  benefits 
justify  the  increased  costs  or  if  other  program  costs  can  be  traded  off. 

-  Structure  an  optimal/near  optimal  ADS-enhanced  testing  program  based  on  the 
appropriate  balance  between  cost  savings  and  qualitative  benefits. 

The  following  subsections  describe  each  of  the  methodology  steps  in  more  details. 

B.2.1  Appropriate  Use  of  ADS 

The  general  role  of  ADS  in  PGM  T&E  was  discussed  in  Section  3.2.3,  and  Table  1  from  that 
section  identified  the  various  performance  evaluation  techniques  used  for  PGM  T&E,  including 
ADS-supported  tests.  Although  the  discussion  in  that  section  focused  on  the  role  of  ADS  in 
PGM  DT&E  and  OT&E,  ADS  can  support  the  other  testing  phases,  including  early  operational 
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assessment  (EOA),  operational  assessment  (OA),  initial  operational  test  and  evaluation  (IOT&E), 
and  follow-on  operational  test  and  evaluation  (FOT&E).  This  is  illustrated  in  Figure  B- 1 . 


nominal  time  (years) 


T&E 

METHODS 


SYSTEM 

ACQUISITION 

PHASES 


T&E 

PHASES 


(0-2) 


(2-4) 


(4-7) 


(10-50) 


Launches  (Live  Firej  Tests) 


Captive-Carry  Tests; 


ADS  using  PGM  HWlL 


HWIL  Lab 


Lab/Field  Tests 


ADS  using  PGM  D$M 


Analytical  (DSM) 


Analytical  Study 


MSS  0 


MSI 


Concept 

Exploration 


MS  II 


Program 
Definition  &  J 
Risk  Reduction  1 


Engineering  & 
Manufacturing 
Development 


m$  m 


component 


DT&E  ‘ 

subsystem 


®  EOA  ■ 


»OA 


system 

IOT&E ■ 


Production,  Fielding/ 
Deployment,  & 
Operational  Support 


FOT&E ■ 


Figure  B-l.  Role  of  ADS-Based  Testing  During  PGM  System  Acquisition  and  Testing  Life 

Cycle. 


As  Figure  B-l  shows,  a  PGM  DSM  normally  becomes  available  during  the  initial  system 
acquisition  phase,  so  that  ADS-based  evaluations  which  use  a  PGM  DSM  can  begin  during  this 
phase  (e.g.,  for  requirements  development),  even  before  formal  T&E  begins.  As  soon  as 
prototype  PGM  hardware  is  developed  during  the  second  acquisition  phase,  all  T&E  methods  can 
begin  to  be  used,  as  appropriate.  As  system  acquisition  proceeds,  all  T&E  methods  can  continue 
to  be  used  to  test  the  evolving  PGM  system  and  subsystems.  The  optimal  use  of  the  various 
methods  depends  on  the  PGM  performance  areas  being  evaluated,  as  Table  2  in  Section  3.2.3 
shows.  Note  from  Figure  B-l  that  ADS-based  testing  can  be  used  throughout  the  system 
acquisition  and  testing  life  cycle  as  PGM  simulation  resources  are  developed  and  refined,  and 
considerations  for  applying  ADS  to  PGM  T&E  are  discussed  in  Section  4. 
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B.2.1.1  ADS  Benefits 


In  analyzing  the  appropriate  use  of  ADS  in  the  total  testing  program,  the  potential  benefits  of 
adding  ADS  should  be  evaluated.  General  benefits  were  discussed  in  Section  3.2.2  and  fall  into 
the  categories  of  cost  savings,  improved  testing,  and  more  efficient  testing.  Potential  cost 
savings  are  determined  in  the  next  step,  but  any  benefits  from  improved  or  more  efficient  testing 
should  be  considered  before  proceeding.  These  are  repeated  from  Section  3.2.2  and  expanded. 

-  Improved  testing  benefits. 

--  Testing  using  a  linked  laboratory  ADS  architecture  is  more  reproducible  than  live  fire 
testing,  because  scenario  conditions  are  more  readily  controlled  and  trials  can  be 
replayed  for  additional  PGM  responses.  This  allows  more  trials  to  be  combined  for 
analysis,  giving  greater  confidence  in  evaluation  results. 

—  Testing  using  manned  shooter  and  target  simulators  linked  to  a  PGM  DSM  or  HWIL 
laboratory  can  be  used  to  realistically  evaluate  man-in-the-loop  reactive  CM.  This 
cannot  be  done  in  live  fire  testing  due  to  obvious  safety  constraints. 

—  ADS-supported  testing  allows  the  evaluation  of  certain  classified  techniques  in  which 
the  ECM  device  cannot  be  permitted  to  radiate  its  RF  emission  on  an  open  range. 
Rather,  the  ECM  emissions  can  be  restricted  to  the  PGM  HWIL  laboratory  where  they 
are  screened  from  unauthorized  observation  and  where  the  effects  of  the  ECM  on 
PGM  performance  can  be  immediately  observed  by  analysts. 

—  ADS  allows  the  force  density  of  the  scenario  to  be  increased.  The  number  of  friendly 
and  threat  systems  can  be  increased  by  representing  them  with  either  manned 
laboratories  (if  realistic  man-in-the-loop  control  of  the  systems  is  needed)  or  DSMs  (if 
scripted  behavior  is  acceptable).  The  inability  to  evaluate  system  performance  in 
combat-representative  environments  is  a  common  limitation  in  OT&E  and  an  area  in 
which  ADS  can  improve  the  operational  test  (OT)  environment. 

--  ADS-supported  tests  exhibit  more  realism  than  either  analytical  simulation  models 
(because  actual  hardware  is  used)  or  standalone  HWIL  laboratories  (because  realistic 
shooter  and  target  inputs  are  provided). 

—  The  live  shooter-target  ADS  architecture  can  be  used  for  realistic  engagement  tactics 
development  and  evaluation. 

-  More  efficient  testing  benefits. 

—  Shooter/PGM  integration  can  be  effectively  evaluated  using  ADS  configurations. 

—  Integration  check  out  can  begin  early  in  the  acquisition  cycle  before  a  complete 
PGM  system  is  available  by  using  the  linked  laboratory  ADS  configuration  and 
including  only  key  PGM  subsystems  in  the  PGM  HWIL  laboratory.  Note  that  this 
same  configuration  can  also  be  used  to  verify  the  integration  of  a  new  shooter 
platform  to  an  existing  PGM  system. 

—  If  the  PGM  receives  data  link  support  from  the  shooter  during  its  flyout,  the  live 
shooter-target  ADS  configuration  can  be  used  to  accurately  evaluate  the  quality  of 
the  support  messages. 

-  Testing  using  a  live  shooter-target  ADS  architecture  is  more  efficient  than  live  fire 
testing  because  the  analysts  get  immediate  feedback  on  each  trial  of  a  multiple  trial 
mission.  This  allows  adjustments  to  be  made  to  the  remaining  test  matrix,  if 
necessary,  while  the  live  shooter  and  target  platforms  are  still  on  range.  This 
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“analyst-in-the-loop”  feature  of  ADS  testing  would  be  especially  useful  in  efficiently 
progressing  through  an  ECM  testing  matrix  which  involves  varying  a  number  of 
ECM-related  parameters.  (Up  to  15  trials  were  executed  during  each  two-hour  SIT 
LFP  mission.) 

-  Live  fire  tests  can  be  realistically  rehearsed  using  ADS.  This  would  ensure  the  proper 
setup  of  the  scenario  and  reduce  wasted  live  fire  attempts  in  which  the  proper  scenario 
conditions  are  not  achieved  or  would  result  in  anomalies  (i.e.,  cost  avoidance).  This 
use  of  ADS  would  also  reduce  the  risk  of  a  live  fire  testing  program  by  identifying 
scenarios  which  cannot  be  correctly  executed  or  which  cannot  achieve  the  stated 
objectives. 

Also,  ADS  implementation  may  benefit  other  parts  of  the  PGM  acquisition  program  besides 
testing.  ADS  using  a  DSM  to  represent  the  PGM  may  aid  in  requirements  development,  and 
resources  developed  during  ADS  implementation  may  be  useful  for  training.  Such  benefits  of 
ADS  implementation  which  are  beyond  the  normal  scope  of  testing  should  also  be  considered  in 
this  determination. 

B.2.1.2  ADS  Implementation  Constraints 

Constraints  and  limitations  in  applying  ADS  must  be  considered  when  determining  the 
appropriate  use  of  ADS-based  testing.  These  include  any  constraints  in  using  the  candidate  PGM 
simulation  being  considered  for  linking  and  any  ADS-related  constraints,  as  discussed  in  Section 
3.3.  The  result  of  considering  these  constraints  will  be  a  determination  of  which  PGM  test 
scenarios  and  conditions  can  be  adequately  evaluated  using  ADS-based  testing  (i.e.,  which 
scenarios/conditions  are  viable  candidates  for  ADS-based  testing). 

B.2.2  Test  Program  Costing 

As  discussed  in  Section  3.2.2,  the  benefits  of  ADS-supported  testing  are  best  realized  when  this 
technique  is  added  to  a  total  PGM  testing  program  and  is  used  to  supplement,  rather  than  replace, 
other  testing  techniques.  Therefore,  a  total  testing  program  should  be  developed  with  and 
without  ADS-supported  testing.  Major  cost  elements  for  each  testing  technique  are  as  follows: 

Non-recurring  costs.  Each  testing  technique  will  require  development  and  verification 
before  PGM  testing  can  begin.  The  cost  of  implementing  ADS-based  testing  can  be 
significantly  reduced  if  ADS  is  planned  for  as  the  other  testing  resources  are  developed 
(so  that  no  major  modifications  of  other  resources,  such  as  HWIL  laboratories,  will  be 
needed  for  ADS  implementation  and  interface  costs  will  be  minimized). 

-  Live  fire  tests.  The  most  significant  recurring  costs  are  for  the  PGM,  target  drone,  and 
test  range  support. 

-  Captive-carry  tests.  The  main  recurring  cost  is  for  test  range  support,  including  use  of  the 
shooter  and  target  platforms  and  the  captive-carry  PGM  pod. 

-  Standalone  HWIL  tests.  The  main  recurring  cost  is  for  HWIL  facility  operation. 

-  ADS  tests  using  live  shooter-target  configuration.  The  recurring  costs  combine  the  cost 
of  a  captive-carry  mission  and  standalone  HWIL  tests  with  the  cost  of  any  additional  real¬ 
time  processing  of  the  data  transferred  from  the  live  platforms  to  the  PGM  HWIL 
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laboratory.  As  a  result,  this  type  of  ADS  test  will  usually  cost  more  than  just  a  captive- 
carry  mission  combined  with  a  standalone  HWIL  test  in  which  data  captured  from  the 
captive-carry  mission  are  replayed  to  drive  the  HWIL  laboratory  post-mission. 

ADS  test  using  linked  HWIL  laboratories.  The  recurring  costs  will  be  the  sum  of  the 
costs  for  operating  each  laboratory  plus  the  cost  of  any  additional  real-time  processing  of 
the  transferred  data. 

The  cost  elements  are  combined  with  proposed  testing  programs  which  either  implement  or  do 
not  implement  ADS-based  testing  in  order  to  determine  the  total  costs  of  the  two  candidate 
programs  (with  and  without  ADS). 

B.2.2.1  Cost  Savings  Case 

The  largest  potential  cost  savings  occur  when  ADS-based  tests  are  used  to  replace  some  of  the 
live  fire  tests.  Typically,  the  cost  of  the  PGM  and  the  target  drone  is  much  more  than  the  range 
support  cost  (e.g.,  the  AMRAAM  missile  and  QF-106  drone  cost  about  $550,000,  compared  to 
about  $50,000  for  range  support).  The  replacement  of  a  live  fire  test  with  an  ADS-based  test 
essentially  saves  the  cost  of  the  PGM  and  target,  since  the  range  support  cost  for  the  live  test  is 
nearly  the  same  as  the  recurring  cost  of  the  ADS  test. 

However,  there  are  limitations  in  eliminating  live  fire  tests,  because  public  law  requires  some 
live  PGM  testing  and  because  there  is  usually  a  minimum  number  of  live  shots  needed  to 
evaluate  the  PGM  reliability.  If  the  number  of  live  shots  is  already  at  the  minimum  needed  for 
the  reliability  evaluation,  then  none  of  the  live  shots  can  be  replaced  with  ADS-based  tests.  In 
this  case,  the  blending  of  ADS  into  the  PGM  testing  program  will  normally  result  in  increased 
cost  (unless  the  ADS-based  tests  can  replace  some  of  the  non-ADS  simulation  testing  at  no 
additional  cost),  and  the  benefits  of  using  ADS-based  tests  must  be  carefully  evaluated  against 
the  increased  costs. 

If  some  of  the  live  fire  tests  can  be  replaced  with  ADS-based  tests  without  impacting  other 
testing  requirements  (e.g.,  the  reliability  determination),  then  the  maximum  allowable  number  of 
replacements  is  determined.  The  test  matrix/scenarios/conditions  are  then  examined  to 
determine  which  of  these  are  best  accomplished  using  live  fire  tests  and  which  can  be  adequately 
tested  using  ADS  (see  Section  B.2.1.1),  up  to  the  maximum  allowable  number  of  replacements. 
Note  that  each  live  fire  test  can  only  evaluate  a  single  scenario,  but  that  each  ADS-based  test 
mission  can  usually  evaluate  multiple  scenarios  (e.g.,  during  LFP  testing,  a  single  two-hour 
mission  consisted  of  about  15  separate  passes,  each  of  which  could  have  used  a  separate 
profile/scenario)  or  repeat  the  same  scenario  multiple  times  for  added  confidence.  Hence,  there 
can  be  considerable  leverage  in  replacing  live  shots  with  ADS-based  tests  (more  test 
conditions/trials  can  be  evaluated  at  a  lower  cost). 

B.2.2.2  No  Cost  Savings  Case 

For  testing  programs  in  which  no  live  fire  tests  can  be  eliminated,  it  may  be  possible  to  add  ADS- 
based  testing  without  increasing  the  total  testing  costs.  This  would  be  done  by  replacing  some  of 
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the  tests  using  standalone  simulation  techniques  with  ADS-supported  tests.  As  discussed  in 
Section  3.2.3,  there  are  some  cases  in  which  ADS-supported  tests  allow  PGM  performance  areas 
to  be  evaluated  which  could  not  be  evaluated  using  the  PGM  simulation  in  a  standalone 
configuration.  This  is  especially  true  for  those  cases  in  which  it  is  necessary  to  provide  man-in- 
the-loop  shooter  and  target  inputs  to  the  PGM  simulation. 


In  general,  each  ADS-supported  test  will  cost  more  than  the  corresponding  standalone  simulation 
test.  Thus,  in  order  to  keep  the  total  testing  costs  constant,  it  will  be  necessary  to  only  replace  a 
fraction  of  the  standalone  tests,  and  the  ADS-supported  test  cases  must  be  carefully  selected  to  be 
only  those  which  require  linking  for  valid  results. 

Even  in  those  cases  in  which  some  live  fire  tests  can  be  replaced  with  ADS-based  tests,  the 
preference  may  be  to  keep  the  total  testing  costs  constant  by  adding  the  appropriate  number  of 
ADS-based  missions  to  replace  each  live  test.  This  would  allow  many  more  test 
scenarios/conditions  to  be  evaluated  during  the  testing  program.  For  example,  results  from  the 
SIT  LFP  show  that  when  the  cost  of  an  AMRAAM  missile  and  a  QF-106  drone  are  considered,  a 
single  live  AMRAAM  live  fire  test  could  be  replaced  by  about  ten  ADS-based  tests  which  use 
the  LFP  ADS  architecture  at  no  additional  cost.  Further,  each  of  these  two-hour  ADS  missions 
can  normally  execute  about  15  passes.  Hence,  for  this  example,  it  would  be  possible  to  trade  off 
one  live  fire  test  profile  against  150  ADS-supported  test  profiles.  The  result  would  be  greater 
confidence  in  the  results  of  the  PGM  evaluation. 

B.2.2.3  Cost  Increase  Case 

When  ADS-based  tests  are  merely  added  to  a  total  testing  program  without  replacing  any  of  the 
other  testing  techniques,  then  the  total  testing  costs  will  increase.  In  this  case,  as  in  the  “no  cost 
increase  case,  it  is  necessary  to  carefully  select  test  scenarios/conditions  for  ADS 
implementation  and  to  thoroughly  evaluate  the  added  benefits  of  using  ADS  in  the  testing 
program  versus  the  increased  costs.  A  potential  benefit  which  should  be  considered  by  the  PGM 
program  manager  is  cost  avoidance  due  to  reduced  risk.  If  improved  testing  in  any  program 
phase  would  result  in  a  significant  reduction  of  the  risk  of  redesign  or  redevelopment,  the 
program  manager  could  trade  off  the  cost  avoidance  against  the  investment  in  ADS-based  testing. 

Using  ADS-based  testing  would  not  necessarily  result  in  an  increase  in  the  total  cost  of  the  PGM 
acquisition  program  in  this  case.  Because  of  the  benefits  of  adding  ADS-based  testing,  the  PGM 
program  manager  should  consider  trading  off  other  program  costs  (e.g.,  those  for  requirements 
definition  or  training,  since  ADS  configurations  may  be  able  to  assist  in  these  areas)  in  order  to 
implement  ADS-based  testing  without  increasing  the  total  program  cost. 

B.2.3  Optimal  ADS-Enhanced  Testing  Program 

After  completing  the  analysis  in  the  previous  methodology  step,  it  should  be  straightforward  to 
construct  the  optimal  ADS-enhanced  testing  program.  The  resulting  program  would  be  justified 
by  cost  savings,  qualitative  benefits,  or  a  combination  of  these. 
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B.3.0  Cost  Benefit  Examples 
B.3.1  Cost  Elements  Example 

Cost  data  were  collected  during  the  SIT  LFP  for  the  costs  of  implementing  the  LFP  ADS 
architecture,  conducting  AMRAAM  testing  using  this  architecture,  and  conducting  AMRAAM 
testing  using  traditional,  non-ADS  testing  techniques.  The  approximate  costs  for  executing  each 
testing  technique  are  as  follows  (test  planning  and  data  analysis  costs  are  not  included): 

-  Live  fire  test  cost:  about  $600,000,  including  the  cost  of  the  AMRAAM  and  target  drone 
($600,000  per  pass). 

Captive-carry  flight  test  cost:  about  $24,000  for  a  two-hour  mission  which  can  perform 
1 5  passes  ($  1 600  per  pass). 

Standalone  HWIL  mission  cost:  about  $7,000  for  a  two-hour  mission  which  can 
nominally  perform  25  runs  ($280  per  run). 

-  LFP  ADS  mission  cost:  about  $44,000  for  a  two-hour  mission  which  achieves  15  passes 
(-$2900  per  pass). 

-  LFP  ADS  implementation  cost:  about  $2  million,  including  the  cost  of  developing  the 
AASI  special-purpose  interface,  modifying/upgrading  Eglin  range  capabilities  and 
facilities,  and  integration  testing.  Note  that  a  portion  of  the  cost  was  for  upgrading  Eglin 
range  capabilities  (e.g.,  TDP  improvements)  and  that  these  costs  would  not  normally  be 
borne  by  the  PGM  testing  program. 

Cost  data  are  also  available  for  the  linked  laboratory  ADS  configuration,  as  implemented  in  the 
SIT  LSP.  In  comparing  to  LFP  costs,  it  should  be  remembered  that  the  LSP  configuration 
included  an  AIM-9  Sidewinder,  rather  than  an  AIM- 120  AMRAAM,  HWIL  laboratory. 

LSP  ADS  mission:  about  $20,000  for  a  four-hour  mission  which  achieves  80  runs 
(-$250  per  run). 

-  LSP  ADS  implementation  cost:  about  $  1 .2  million,  including  the  cost  of  modifying  Point 
Mugu/China  Lake  facilities  and  integration  testing.  This  cost  reflected  the  need  to  retrofit 
the  existing  facilities  so  that  they  could  be  linked.  If  the  requirement  for  linking  were 
planned  into  the  facilities  design,  it  is  believed  that  the  cost  of  implementing  linking 
would  have  been  significantly  lower. 

B.3.2  Testing  Program  Cost  Example 

As  an  example  of  estimating  the  cost  of  a  testing  program  with  and  without  ADS-based  testing,  a 
live  fire  test  program  without  ADS  is  compared  to  a  test  program  in  which  (1)  some  of  the  live 
tests  are  replaced  with  ADS-based  tests  and  (2)  an  ADS  configuration  is  used  to  rehearse  the  live 
tests. 
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For  the  specific  example,  the  AMRAAM  FOT&E[3A]  test  plan3  was  examined  to  determine  the 
maximum  number  of  launch  profiles  for  which  the  SIT  LFP  ADS  architecture  could  provide 
valid  data.  Criteria  for  selecting  candidate  profiles  included  F-15  or  F-16  shooter,  no  multiple 
shooters  with  simultaneous  launches,  fighter  drone  targets,  resolved  targets  (if  multiple  targets 
were  involved),  no  target  beam  maneuvers  during  missile  intercept,  and  no  chaff  CM.  Out  of  29 
live  profiles,  three  met  the  screening  criteria  and  were  viable  candidates  for  ADS 
implementation.  In  other  words,  a  maximum  of  three  FOT&E[3A]  profiles  could  be  performed 
using  the  LFP  ADS  configuration,  rather  than  live  fire.  For  each  live  fire  profile  replaced  with  an 
ADS  mission,  the  cost  savings  is  about  $550,000.  Thus,  about  $1.65  million  could  be  saved  if  all 
three  live  profiles  were  performed  with  ADS  missions.  (Note  that  even  though  each  ADS 
mission  can  include  15  passes,  it  was  not  possible  to  combine  all  three  live  profiles  into  a  single 

ADS  mission.  This  was  because  of  differences  in  shooters  and  other  key  scenario  features  in  the 
three  profiles.) 

The  live  fire  profiles  in  FOT&E[3A]  were  rehearsed  using  captive-carry  missions.  Data 
collected  from  the  captive-carry  missions  were  normally  used  after  the  mission  as  inputs  into  an 
AMRAAM  DSM  for  the  purpose  of  verifying  the  launch  profiles  and  missile  flyouts.  If  this  were 
done  using  the  LFP  ADS  configuration  instead,  the  MISILAB  HWIL  laboratory  would  provide 
higher  fidelity  missile  flyout  results  during  the  mission,  allowing  real-time  verification  and 
refinement  of  the  live  fire  profiles.  For  each  captive-carry  mission  replaced  by  the  ADS 
configuration,  the  additional  cost  is  about  $20,000.  Assuming  all  three  live  profiles  which  could 
be  replaced  by  ADS  were  indeed  replaced  and  the  26  remaining  live  profiles  were  all  rehearsed 
using  the  ADS  LFP  configuration,  the  additional  cost  for  the  rehearsals  would  be  about  $460,000 
(three  profiles  which  would  have  been  rehearsed  using  captive-carry  missions  were  eliminated, 

saving  about  $60,000),  and  the  net  cost  savings  from  reducing  the  number  of  live  shots  would  be 
about  $1.2  million. 

Note  that  the  non-recurring  implementation  cost  for  the  LFP  ADS  architecture  was  about  $2 
million.  When  this  investment  cost  is  also  considered,  the  maximum  use  of  ADS  in  support  of 
FOT&E[3A]  would  have  cost  an  additional  $350,000  if  ADS  were  only  used  to  replace  three  of 
the  live  profiles  and  an  additional  $870,000  if  ADS  were  also  used  to  rehearse  all  live  profiles. 

This  example  illustrates  that  if  ADS  is  only  used  to  support  one  phase  of  testing  involving  a 
relatively  small  number  of  live  fire  tests,  cost  savings  may  not  be  possible.  If  ADS-based  testing 
is  instead  implemented  over  all  phases  of  testing,  cost  savings  would  be  expected.  For  example, 
the  analysis  of  the  FOT&E[3A]  profiles  showed  that  about  10%  of  them  could  be  replaced  with 
ADS-based  tests.  If  this  same  percentage  is  applied  to  a  total  missile  testing  program  consisting 
of  100  live  shots,  then  the  number  which  could  be  replaced  would  be  ten.  In  this  case  (assuming 
the  same  costs  as  for  AMRAAM  FOT&E[3A]  and  the  SIT  LFP),  there  would  be  a  cost  savings  of 
about  $1.9  million  (after  accounting  for  the  $2  million  ADS  implementation  cost)  if  the  ADS 
configuration  was  also  used  to  rehearse  all  90  live  profiles  and  a  cost  savings  of  about  $3.7 
million  if  the  ADS  configuration  was  not  used  for  rehearsal.  The  increased  cost  savings  with 
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each  replaced  live  shot  is  illustrated  in  Figure  B-2  (for  the  case  of  not  using  the  ADS 
configuration  for  rehearsal).  Note  that  the  break-even  number  of  reduced  live  shots  is  four  (i.e., 
this  number  of  reduced  shots  is  required  to  recover  the  $2  million  ADS  development  cost). 

The  examples  in  this  section  are  meant  only  to  illustrate  the  application  of  the  cost  benefit 
analysis  methodology  and  should  not  be  considered  as  general  results.  Each  specific  PGM 
program  must  be  analyzed  to  determine  the  cost  effectiveness  of  ADS  implementation.  In 
particular,  if  ADS  implementation  is  properly  planned  for  early  in  the  acquisition  and  testing 
program,  the  cost  of  implementation  may  well  be  significantly  lower  than  the  $2  million  ADS 
development  cost  in  these  examples.  (Note  that  implementation  costs  are  indeed  strongly 
dependent  on  the  specific  ADS  architecture  —  the  SIT  LSP  development  cost  was  about  $1.2 
million  versus  about  $2  million  for  the  SIT  LFP.) 

B.4.0  Conclusion 

A  general  methodology  has  been  presented  for  analyzing  the  cost-effectiveness  of  implementing 
ADS-based  testing  in  a  PGM  testing  program.  The  application  of  this  methodology  should 
always  be  done  in  a  total  program  sense  in  which  ADS  is  blended  into  the  entire  PGM 
acquisition  and  testing  program.  If  the  strengths  and  benefits  of  ADS-based  testing  are  properly 
exploited,  it  should  be  possible  to  design  an  improved,  more  thorough,  and  higher  confidence  test 
and  evaluation  program,  while  still  remaining  within  budgetary  constraints. 
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100  Live  Shot  Program  -  Millions  of  $ 
(tradeoff  Live  Shot  for  ADS  mission) 


Figure  B-2.  Levels  of  Program  Costs/Savings  with  Varying  Numbers  of  ADS  Replacement  Missions. 
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ADS  IMPLEMENTATION  GUIDELINES 


This  appendix  outlines  the  steps  in  implementing  ADS-based  testing  in  which  a  shooter  entity, 
target  entity,  and  PGM  entity  are  linked.  It  follows  the  steps  given  in  the  HLA  Federation 
Development  and  Execution  Process  (FEDEP)  model  [Ref.  C-l].  In  comparing  these  guidelines 
with  the  FEDEP  model  note  that  the  term  “ADS  architecture”  used  here  equates  to  the  term 
“federation”  in  the  FEDEP  model,  and  the  terms  “facilities”  and  “live  entities”  used  here  equate 
to  the  term  “federates”  in  the  FEDEP  model. 

STEP  1 :  Requirements  Definition. 

Activity  1.1:  Sponsor  Needs  Identification. 

-  Describe  critical  systems  of  interest. 

-  Determine  resources  which  will  be  available  to  support  ADS  implementation  (e.g., 
funding,  personnel,  tools,  facilities). 

-  Determine  any  known  test  constraints  (e.g.,  due  dates,  security  requirements). 

Activity  1.2:  Objectives  Development. 

-  Determine  test  objectives,  scenarios,  conditions,  and  which  PGM  performance  parameters 
(e.g.,  measures  of  effectiveness)  are  to  be  evaluated. 

-  Develop  an  ADS  architecture/network  development  plan  including  approximate  schedule 
and  major  milestones. 

-  Develop  a  configuration  management  plan. 

-  Determine  operational  context  constraints  or  preferences,  including  geographical  regions, 
environmental  conditions,  threats,  and  tactics. 

-  Identify  security  requirements,  including  estimated  classification  level  and  possible 
designated  approval  authority  or  authorities. 

-  Develop  initial  verification,  validation,  and  accreditation  (VV&A)/test  plans. 

-  Select  tools  to  support  scenario  development,  conceptual  analysis,  VV&A  and  test 
activities,  and  configuration  management. 

STEP  2:  Conceptual  Model  Development. 

Activity  2.1:  Scenario  Development. 

-  Identify  major  entities  that  must  be  represented. 

-  Describe  capabilities,  behavior,  and  relationships  between  major  entities  over  time  (e.g., 
shooter  and  target  profiles  and  shooter  support  of  PGM). 

Specify  relevant  environmental  conditions  which  impact  or  are  impacted  by  the  entities. 
Activity  2.2:  Conceptual  Analysis. 

-  Evaluate  entity  characteristics,  relationships,  and  interactions. 

Determine  if  ADS  implementation  is  appropriate. 

-  In  general,  ADS  implementation  is  necessary  if  entities  interact  with  each  other  and  if 
linking  is  necessary  to  permit  the  interactions. 

-  Determine  appropriate  representation  for  each  entity. 

-  Choice  of  appropriate  representation  depends  on  the  availability  of  simulation 
facilities,  the  capabilities  of  range  instrumentation  and  data  processing,  the  nature  of 
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entity  interactions  (e.g.,  whether  or  not  the  target  reacts  to  the  PGM),  and  the 
objective  of  the  testing  (see  Section  4  in  main  report). 

-  Choice  depends  on  requirements  and  must  be  determined  in  parallel  with  a  detailed 
requirements  determination.  For  example,  live  entities  may  not  have  sufficiently 
accurate  TSPI  (due  to  range  instrumentation  and  processing  limitations),  or 
sufficiently  small  latencies  (due  to  data  processing  times),  or  feedback  from  the  PGM 
simulation  to  the  live  target  may  not  be  possible. 

Choice  depends  on  degree  of  realism  (fidelity)  needed:  Live  shooter/target  provide 
more  realistic  inputs  to  the  PGM  simulation  than  virtual  (manned  lab)  shooter/target; 
PGM  HWIL  provides  more  realistic  PGM  performance  than  PGM  DSM. 

Choice  of  DSM  or  HWIL  for  PGM  also  depends  on  PGM  performance  areas  to  be 
evaluated  (see  Table  2  in  main  report). 

Determine  ADS  architecture  requirements. 

—  Data  requirements. 

Determine  data  types  which  must  be  exchanged  among  entities  to  permit  interact. 
—  Entity  state  data 

Data  exchanged  between  shooter  and  PGM  before  launch. 

—  Data  provided  by  shooter  to  PGM  after  launch. 

—  PGM  launch  indication. 

—  PGM  detonation  indication. 

Target  CM  (e.g.,  flares,  smoke,  ECM)  deployment  indication. 

- —  Trial  start  and  stop  notification. 

—  Determine  data  required  to  achieve  test  objectives. 

Data  exchanged  among  entities  and  recorded  at  multiple  locations.  Identify 
data  source  and  each  required  recording  location.  Document  with  interface 
control  document. 

—  Data  not  exchanged  but  recorded  at  originating  node. 

—  Data  required  for  test  control. 

—  Determine  rates  of  data  exchanged  among  entities  (in  bits  per  second). 

— —  Determine  if  data  will  be  sent  out  of  the  simulation  interface  at  the  rate  it  is 
received  by  the  generating  entity  or  if  dead  reckoning  will  be  used  to  reduce 
data  rates  over  the  wide  area  network  (WAN)  (because  of  WAN  bandwidth 
restrictions). 

—  Determine  TSPI  accuracy  and  smoothness  requirements  for  live  entities.  This 
depends  on  test  objectives  and  on  which  PGM  parameters  are  being  evaluated. 

—  Determine  requirement  for  data  time  stamp  accuracy. 

—  Determine  classification  of  data  and  security  handling  requirements. 

--  Latency  requirements. 

—  Acceptable  latency  and  latency  variations. 

—  Closed-loop  interaction  requirements. 

--  Data  quality  requirements. 

—  Acceptable  level  of  ADS-induced  errors  (e.g.,  dropout  rate,  missing  PDUs). 

--  Synchronization  requirements. 

—  Entity  state  data  synchronization. 

—  Shooter  support  message  synchronization  to  entity  state  data. 
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—  Determine  if  any  other  data  types  must  be  synchronized  for  input  to  receiving 
entity. 

Real-time  data  processing  requirements. 

—  Determine  processing  necessary  to  achieve  TSPI  accuracy  (for  live  shooter/target) 
and  synchronization  requirements  and  to  compensate  for  latency  effects  (if 
possible). 

Post-test  data  management  requirements. 

—  Determine  data  sources  and  amounts. 

—  Determine  data  handling  requirements. 

—  Determine  data  storage  requirements  (based  on  data  requirements)  at  each  site,  as 
well  as  central  data  storage. 

Data  analysis  requirements. 

—  Determine  analysis  approach  for  all  types  of  data. 

—  Identify  commercial  or  government  off-the-shelf  statistical  analysis  tools  and 
other  post-processing  tools  to  be  used  for  data  analysis/reduction. 

—  Determine  requirements  for  any  custom  data  analysis/processing  tools. 

Network  requirements. 

—  Determine  protocols  to  be  used. 

—  Decide  if  standard  protocols  (e.g.,  DIS  PDUs)  are  to  be  used  or  if  it  would  be 
advantageous  to  keep  data  in  formats  generated  by  entities.  If  transmitting  and 
receiving  entities  use  the  same  coordinate  system  (which  is  different  than  the 
geocentric  DIS  coordinate  system),  processing  time  and  coordinate 
transformation  errors  would  be  reduced  by  not  using  DIS  PDUs. 

—  Identify  prelaunch  data  exchanged  between  the  shooter  and  PGM  (e.g.,  SMS 
data)  which  is  best  kept  in  its  tactical  protocol. 

—  Determine  if  HLA  will  be  implemented.  The  use  of  HLA  will  affect  the  choice  of 
protocol  and  simulation  interface  design  and  requires  an  appropriate  RTI. 

—  Determine  if  data  from  node  will  be  broadcast  or  transmitted  point-to-point.  HLA 
implementation  is  a  factor  in  this  determination. 

—  Determine  data  encryption  requirements  based  on  classification  of  data  to  be 
passed  over  WAN. 

Interface  requirements. 

—  Simulation  interfaces. 

- —  Determine  coordinate  transformation  requirements  to  convert  from  coordinate 
frame  of  transmitting  entity  to  coordinate  frame  of  data  protocol  used  over 
WAN  (e.g.,  DIS  PDUs)  and  to  convert  from  coordinate  frame  of  data  protocol 
used  over  WAN  to  coordinate  frame  of  receiving  entity. 

—  Determine  dead  reckoning  requirements.  Determine  if  dead  reckoning  will  be 
used  at  the  transmitting  node,  as  well  as  at  the  receiving  node.  Determine  if 
second  order  or  first  order  dead  reckoning  is  required  based  on  position  and 
orientation  accuracy  requirements  and  estimated  latencies. 

—  Determine  requirements  for  simulation  interfaces  to  synchronize  received  data 
to  receiving  entities.  May  require  simulation  interface  to  dead  reckon  entity 
state  data  to  receiving  simulation  frame  rate. 

—  Special-purpose  interfaces. 
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—  Determine  if  special  interfaces  will  be  required  to  transfer  synchronized 
shooter-generated  data  to  the  PGM  simulation.  Such  data  may  be  required  to 
initialize  and  launch  the  PGM  and  to  provide  it  with  targeting  updates  during 
its  flyout. 

—  Test  control  and  monitoring  requirements. 

—  Develop  test  control  concept. 

—  Determine  central  control  location. 

—  Determine  test  coordinator  and  location  at  each  distributed  node/facility. 

—  Determine  live  entity  control  technique. 

Determine  if  local  range  control  is  required  (due  to  safety  considerations  or 
policy). 

If  remote  live  entity  control  is  possible,  determine  monitoring/ 
communications  requirements. 

—  Determine  display  and  monitoring  requirements. 

—  Determine  requirements  for  monitoring  status  of  entities. 

-—Determine  requirements  for  monitoring  status/performance  of  distributed 
network. 

Determine  voice  communications  requirements  and  adequacy  of  existing 
telephone  systems.  ' 

STEP  3:  ADS  Architecture  Design 
Activity  3.1:  Design  Architecture 

-  Determine  location  of  nodes. 

--  Select  simulation  facilities  based  on  fidelity,  availability,  cost,  and  schedule. 

For  live  shooter/target  configurations,  select  test  ranges  based  on  instrumentation 
quality  and  quantity,  data  processing  capability,  availability,  cost,  and  schedule. 

-  Conduct  surveys  of  each  site  (node  location). 

--  Determine  facility  communications  architecture  and  requirements. 

Determine  physical  space  requirements  for  tester-supplied  equipment  and  personnel. 

-  Determine  security  approach. 

--  Designate  security  point  of  contact  (POC). 

Perform  security  risk  assessment  and  develop  concept  of  operations. 

-  Determine  WAN  bandwidth  requirement. 

-  Determine  average  and  maximum  aggregate  data  rate  by  adding  rates  from  each  entity 
broadcast  over  the  WAN. 

--  Bandwidth  requirement  is  given  by  the  aggregate  data  rate  plus  a  50%  -  100%  margin 
for  overhead  and  unanticipated  traffic. 

-  Select  WAN/LAN. 

Determine  if  DoD-sponsored  network  will  support  requirements  or  if  commercial 
leased  lines  must  be  used. 

—  Consolidate  network  requirements. 

—  Acceptable  latency  limits. 

—  Aggregate  data  rates. 

—  Network  management/control. 
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—  Submit  requirements  to  Headquarters,  Defense  Information  Systems  Agency  (HQ 
DISA)/D36  for  determination  of  whether  Defense  Information  Systems  Network 
(DISN)  common-user  services  (e.g.,  DSI,  Secure  Internet  Protocol  Router 
Network  (SIPRNET))  will  support  the  requirements  or  if  a  waiver  to  policy  is 
justified  [Ref.  C-2], 

—  If  a  waiver  to  policy  is  justified,  survey  commercial  line  lease  rates. 

—  Determine  if  dedicated  (full-time)  leased  lines  are  required  or  if  on-demand 
leased  lines  will  suffice. 

—  Contract  for  leased  lines. 

Select  network  hardware. 

-  Select  type(s)  of  router(s),  channel  service  units  (CSUs)/data  service  units  (DSUs), 
mulitplexers,  etc. 

—  If  possible,  select  same  type  of  router  for  all  nodes. 

—  Determine  router  addressing  scheme  (see  Ref.  C-3  for  details). 

—  Select  type(s)  of  encryptor(s). 

-  Select  test  control  hardware  and  software. 

--  Determine  communications  requirements. 

—  Determine  data  communications  requirements. 

—  Determine  voice  communications  requirements. 

-  Determine  test  control  center  physical  space  requirements  and  layout. 

-  Determine  data  collection/instrumentation  requirements  based  on  data  requirements. 

--  Select  instrumentation  types  and  data  logging  software. 

—  Determine  instrumentation  locations. 

-  Determine  hardware  requirements  for  data  storage  and  handling  (e.g.,  tape  drives, 
CDs,  optical  disks). 

—  Determine  data  handling  procedures  for  collecting  and  storing  data  from  the 
distributed  network. 

Determine  time  synchronization  method(s)  (e.g.,  method  for  synchronizing  time  stamps 
for  data  loggers  -  see  Refs.  C-3  through  C-5  for  details). 

-  Determine  hardware  required  (e.g.,  IRIG,  GPS). 

-  Determine  software  required  (e.g.,  network  time  protocol  (NTP)). 

Develop  formalized  plan  for  architecture  development  and  integration. 

Activity  3.2:  Develop  Architecture. 

-  Procure  and/or  develop  network  analysis/monitoring  tools. 

--  Determine  required  extent  of  analysis/monitoring. 

—  Troubleshooting  only  versus  collecting  and  analyzing  data. 

—  Bandwidth  monitoring. 

—  All  versus  some  links. 

—  Monitor  local  area  networks  (LANs)  locally  versus  remote. 

—  Hardware  requirements  (using  simple  network  monitoring  protocol). 

—  Communications  monitoring  hardware. 

—  Simulation  monitoring  hardware. 

--  Obtain  permission  (from  appropriate  authority)  to  monitor  or  collect  data. 

-  Develop  procedures  for  secure/encrypted  operations  and  obtain  designated  approval 
authority  approval  for  their  use. 
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Coordinate  security  memoranda  of  agreement  (MOAs)  between  organizations 
involved. 

-  Get  accreditation  for  networks,  facilities,  rooms,  etc. 

--  Get  communications  security  (COMSEC)  account. 

-  Order  keying  material. 

-  Implement  strict  hardware  and  software  configuration  control. 

Develop  data  protocols  to  be  used. 

-  If  HLA  is  to  be  implemented,  a  federation  object  model  (FOM)  must  be  developed 
(see  Section  3.6  of  Ref.  C-l  and  the  DMSO  HLA  website  for  more  details). 

Design  and  build/procure  necessary  interfaces  in  accordance  with  requirements. 

--  Simulation/network  interfaces. 

-  RTI  for  HLA  implementation. 

-  Special-purpose  interfaces. 

—  Interfaces  for  tactical  prelaunch  data  exchanged  between  shooter  and  PGM  (e.g., 
exchange  of  stores  management  system  data). 

—  Synchronization  interfaces. 

-  Determine  and  implement  facility  modifications. 

-  Determine  and  implement  simulation  modifications  necessary  to  utilize  external 
inputs. 

—  Modifications  necessary  to  distribute  incoming  data  to  all  required  hardware 
systems  and  software  programs. 

—  Modifications  necessary  to  input  data  into  simulation  code  in  real  time. 

Modifications  necessary  to  meet  synchronization  requirements. 

---  Modifications  necessary  to  control  PGM  attitude  prior  to  launch. 

Determine  simulation  modifications  necessary  to  generate  required  outputs. 

Determine  and  implement  method  for  achieving  data  time  stamp  accuracy 
requirement. 

-  Determine  and  implement  range  data  processing  modifications  necessary  to  meet 
TSPI  accuracy,  smoothness,  and  latency  requirements. 

Determine  and  implement  facility  modifications  required  for  replay  capability  to  be 
used  during  integration  testing. 

STEP  4:  ADS  Architecture  Integration  and  Test. 

Activity  4. 1 :  Execution  Planning. 

-  Develop  integration  test  plan  which  incrementally  checks  out  configuration  during  build¬ 
up. 

—  Initially  test  each  WAN  link  separately. 

—  First  test  at  CSU/DSU  level  to  make  sure  communications  work  at  lowest  level. 

—  Use  ADS  protocols  to  test  routing. 

—  Use  pings  to  check  for  connectivity  and  loading  problems. 

—  Test  simulation-to-simulation  connections. 

—  Test  simulation-to-simulation  connections  with  all  nodes  on  the  network. 

—  Use  network  analysis/monitoring  tools  to  troubleshoot  the  network. 

—  Use  ADS  protocols  to  test  routing. 

Develop  test  control  procedures. 
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-  Develop  detailed  execution  plans. 

—  For  HLA  implementation,  see  Section  3.7  of  Reference  C- 1 . 

-  For  secure  networks,  develop  security  test  and  evaluation  plan. 

Activity  4.2:  Perform  ADS  Architecture  Integration  and  Testing. 

Install  network  hardware  and  software. 

Perform  compliance  testing. 

—  Test  each  facility/node  individually  to  ensure  that  ADS  capability  and  any  required 
modifications  (including  software)  have  been  correctly  implemented. 

-  Perform  integration  testing. 

—  Check  out  interfaces  and  facility  modifications  with  linking  between  pairs  of  nodes. 

-  Baseline  performance  of  network  with  no  loading  from  the  simulations/entities. 

—  Test  performance  of  critical  portions  of  network  under  loading  representative  of  test 
conditions  to  be  used. 

--  Use  iterative  “test-fix-test”  approaches,  including  replay  of  trials  to  diagnose 
problems  and  verify  fixes. 

-  Perform  risk  reduction  missions. 

-  Execute  scenarios  with  fully  linked  test  execution  configuration  (using  live  entities,  if 
appropriate). 

—  Include  security  certification,  if  required. 

—  Evaluate  test  control  and  monitoring  procedures. 

—  Execute  data  collection  and  analysis  procedures. 

STEP  5:  Execute  and  Analyze  Results. 

Activity  5. 1 :  Test  Execution. 

-  Execute  test. 

--  Develop  test  matrix  based  on  test  objectives. 

—  Execute  test  matrix. 

--  Exercise  test  control/management. 

—  Monitor  execution. 

—  Collect  data. 

--  Maintain  security  posture  of  ADS  architecture  during  execution. 

Activity  5.2:  Results  and  Feedback. 

Analyze  execution  outputs. 

-  Apply  statistical  measures  (if  appropriate)  and  other  data  reduction  methods  to 
transform  raw  data  into  derived  results. 

-  Determine  error  estimates  due  to  inaccuracies  in  measurements  and  sampling. 

-  Use  commercial  or  government  off-the-shelf  statistical  analysis  tools  and  other  post¬ 
processing  tools  for  data  analysis/reduction. 

Perform  feedback. 

-  Use  analysis  results  to  determine  if  all  test  objectives  have  been  met. 

--  If  all  test  objectives  have  not  been  met,  identify  corrective  actions  and  implement 
them,  as  appropriate,  during  re-testing. 

-  Identify  legacy  products  and  make  them  available  to  other  programs. 
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